Model Memory Footprint
The first calculation in almost every AI infra loop: how many bytes does this model occupy, for inference and for training, and does it fit on the card in front of you? Inference is parameters times bytes per parameter (2 in bf16), plus a KV cache that grows with users. Training is 16 bytes per parameter before activations. A 70B model is 141 GB to serve and 1.13 TB to train, and a reader who can produce those two numbers from the parameter count, with the reasoning, has passed the first five minutes of the estimation round.
TL;DR: Inference: weights = parameters × bytes per parameter (bf16 2, fp8 1, int4 0.5), then add the KV cache and about 10% overhead. Training with mixed-precision Adam: 16 bytes per parameter of static state (bf16 weights 2, bf16 gradients 2, fp32 master weights 4, two fp32 optimizer moments 8), then activations on top. Llama 3.1 70B: 141 GB of bf16 weights, so it does not fit one 80 GB H100; 1.13 TB of training state, so it needs sharding across at least 18 cards before a single activation. Say the bytes-per-parameter first, multiply second, sanity-check against the card third.
Where the bytes go
A model in memory is three things: the weights, which are fixed once you pick the model and the precision; the KV cache, which grows with every concurrent user and every token of context; and everything else (activations for the forward pass, the CUDA context, the allocator's fragmentation) that is small at batch 1 and not small at scale. Training adds a fourth: the optimizer's state, which is bigger than the weights themselves.
Inference: the two-line calculation
Weights are parameters times bytes per parameter. The bytes per parameter come from the storage format (Numerics): 4 for fp32, 2 for bf16 or fp16, 1 for fp8 or int8, 0.5 for int4.
Llama 3.1 70B (70.6e9 parameters)
bf16: 70.6e9 × 2 B = 141.2e9 B ≈ 141 GB
fp8: 70.6e9 × 1 B ≈ 71 GB
int4: 70.6e9 × 0.5 B ≈ 35 GB
sanity: an H100 has 80 GB. bf16 does not fit on one card at all; fp8 fits with 9 GB to spare
(too little for a real cache); int4 fits with 45 GB left for KV cache and overhead.
Llama 3.1 8B (8.03e9)
bf16: 8.03e9 × 2 ≈ 16 GB. Fits on one card with 64 GB left.
common slip: "8B in fp16 is 8 GB". The 16 in fp16 is bits, not gigabytes; it is 2 bytes each.
Then the KV cache, which is the number that surprises people. It is computed per token from the architecture and multiplied by context and concurrency; the full derivation is on KV Cache Sizing. The two anchors: Llama 3.1 70B costs 320 KB per token in bf16, so an 8k-token conversation holds 2.6 GB of cache and 32 such conversations hold 84 GB, more than the weights of an 8B model. At long context or high concurrency the cache, not the weights, is what runs out.
Finally the overhead. Reserve about 10% of the card for the CUDA context, the allocator and activations; serving engines expose this as a memory-utilization fraction and default near 90%.
will 70B bf16 serve 32 users at 8k context on 4 × H100?
capacity: 4 × 80 = 320 GB; usable at 90%: 288 GB
weights: 141 GB → free for KV: 288 − 141 = 147 GB
KV per sequence at 8k: 320 KB × 8,192 ≈ 2.6 GB
32 sequences: 32 × 2.6 ≈ 84 GB → fits, with 63 GB of headroom
max concurrency at 8k: 147 ÷ 2.6 ≈ 56 sequences
Training: 16 bytes per parameter, then activations
Mixed-precision training keeps several copies of every parameter, and the reason each exists is worth knowing because the interviewer will ask.
| Tensor | Precision | Bytes per param | Why it exists |
|---|---|---|---|
| weights | bf16 | 2 | what the forward and backward passes multiply with |
| gradients | bf16 | 2 | the backward pass output, same shape as the weights |
| master weights | fp32 | 4 | the optimizer updates a high-precision copy so small updates are not rounded away in bf16 |
| Adam first moment (m) | fp32 | 4 | running mean of gradients |
| Adam second moment (v) | fp32 | 4 | running mean of squared gradients |
| total static | 16 |
70B training state: 70.6e9 × 16 B = 1.13e12 B ≈ 1.13 TB
one H100 holds 80 GB, so at 80% usable (64 GB): 1.13 TB ÷ 64 GB ≈ 17.7 → at least 18 GPUs
just to hold the state, fully sharded (ZeRO-3 / FSDP), before any activation memory.
405B: 405e9 × 16 ≈ 6.5 TB → about 102 GPUs at the same headroom.
Activations sit on top of the static state and depend on the batch, the sequence length and whether recomputation is on. Per transformer layer, with no recomputation, the activations kept for the backward pass are on the order of 34 × s × b × h bytes (from the Megatron "reducing activation recomputation" analysis), where s is sequence length, b micro-batch and h hidden size. For a 70B model (h = 8,192, 80 layers) at s = 8,192 and b = 1 that is 34 × 8,192 × 8,192 ≈ 2.3 GB per layer, 182 GB for the whole model, which is why activation checkpointing exists and why the 16 B/param figure is a floor and not the total.
Fine-tuning changes the arithmetic: LoRA keeps the base weights frozen (2 B/param, no gradients or optimizer state for them) and trains adapters that are 1 to 2% of the model, so a 70B fine-tune fits in roughly 141 GB plus a few GB rather than 1.13 TB. QLoRA quantizes the frozen base to int4 and fits on one card.
Working it in the room
The question arrives as "will X fit on Y?" and the order that scores:
- Say the bytes per parameter for the precision named, out loud, before multiplying. This is where the interviewer learns whether you know why fp16 is 2 bytes.
- Multiply for the weights. Compare with the card. Stop and say whether it fits.
- Say "and then the KV cache", compute it per token from the architecture, multiply by context and users, compare again.
- Reserve the 10%. Give the concurrency number.
- If asked about training, switch to 16 B/param, say why each copy exists, add "plus activations", and give the minimum GPU count.
The answer that sounds right and fails: "70B needs 70 GB". That is int8, and the candidate did not say so.
What to remember
- Bytes per parameter: fp32 4, bf16/fp16 2, fp8/int8 1, int4 0.5. Say it before multiplying.
- 70B bf16 = 141 GB (does not fit an 80 GB card); 8B bf16 = 16 GB; 70B int4 = 35 GB.
- KV cache: 320 KB per token for Llama 3.1 70B in bf16; it is what runs out under load.
- Training: 16 B/param static (2 + 2 + 4 + 4 + 4), so 70B = 1.13 TB and at least 18 H100s before activations; LoRA drops it to about the weights.
- Reserve ~10% of the card; serving engines default to 90% utilization.