How much GPU memory does it take to run Llama 3.1 70B?

TL;DR: Weights alone are parameters × bytes per parameter: 70.6e9 × 2 B = 141 GB in bf16, 71 GB in int8 or fp8, 35 GB in int4. That is before the KV cache and the runtime's working memory, so a bf16 70B does not fit on one 80 GB H100 and needs at least two, and in practice four or eight for any real batch.

How to approach it

Ask two things before writing anything: which precision the model is served in, and whether the interviewer wants weights only or the whole serving footprint. Then say the formula out loud, "parameters times bytes per parameter", and compute the bf16 number first, because it is the reference every other precision is a fraction of. Compare it to a card's memory immediately so the answer has a physical meaning. Close by naming what is not in the number yet: the KV cache and about 10% of runtime overhead.

A strong answer

A typical situation: somebody says a 70B model needs 70 GB, which is the parameter count with a unit attached rather than an arithmetic result, and the deployment is planned around a number that is wrong by a factor of two.

The model memory footprint at rest is one multiplication. The parameter count comes from the model card (Llama 3.1 70B is 70.6 billion, not a round 70), and bytes per parameter comes from the storage format: 2 for bf16 or fp16, 1 for int8 or fp8, 0.5 for int4.

inputs:  N = 70.6e9 parameters
         bytes per parameter: bf16 = 2, int8/fp8 = 1, int4 = 0.5

weights = N × bytes per parameter
  bf16:  70.6e9 × 2   = 141.2e9 B ≈ 141 GB
  int8:  70.6e9 × 1   =  70.6e9 B ≈  71 GB
  int4:  70.6e9 × 0.5 =  35.3e9 B ≈  35 GB

sanity: an H100 has 80 GB, so 141 GB cannot fit on one card in bf16;
        71 GB in fp8 fits on one card with 9 GB to spare, which is not enough for a useful cache;
        35 GB in int4 fits with room for a real batch.

Two habits keep the number honest. Use decimal gigabytes, because that is how vendors quote card memory (80 GB on the H100 is 80e9 bytes), so 141.2e9 bytes is 141 GB and the comparison is like for like. And say the precision every time you say a size: "141 GB in bf16" is an answer, "141 GB" is a guess that happens to be right.

Weights are the floor, not the footprint. A serving process also needs:

• The KV cache, which is the part that scales with users and context. For this model it is 320 KB per token in bf16 (from KV cache sizing), so a single 8k-token sequence adds 2.6 GB and a batch of 32 such sequences adds 84 GB, more than half the weights again.
• Runtime working memory: CUDA context, activation buffers for the current step, the allocator's fragmentation. Budget 5 to 10% of the card.

So the question "does bf16 70B fit on two H100s?" has the answer "the weights do, 141 GB into 160 GB, but only 19 GB is left for the cache and overhead, which is about seven 8k sequences." That is why the common deployment is four or eight cards in tensor parallel, or fp8 weights, which halve the floor and leave the cache room.

serving budget on 8 × H100, bf16 weights:
  capacity           = 8 × 80 GB = 640 GB
  usable at 90%      = 576 GB
  minus weights      = 576 − 141 = 435 GB free for KV cache
  at 8k context      = 435 GB ÷ 2.6 GB per sequence ≈ 167 concurrent sequences
sanity: a 2.6 GB cache per user is the same order as the weights per card (17.6 GB), so
        a few dozen long-context users can outweigh the model itself.

The condition that reverses the "eight cards" default is a latency-insensitive batch workload with short prompts, where int4 weights on one card and a small batch is the cheapest configuration and the quality loss is measured and accepted.

The reversal condition: a quantized deployment. At fp8 the same model is 71 GB and fits on one card with room for a cache, and at int4 it is 35 GB, so the answer to "does it fit" is a different number for each precision and the question is incomplete without one. Model Memory Footprint carries the same arithmetic for every format. Capacity Planning and Utilization is what turns this figure into a fleet, and nvidia-smi --query-gpu=memory.used after load is the check that it held.

What interviewers probe next

• "Why 70.6 and not 70?" The embedding and output matrices are counted in the total; the 0.6 billion is 1.2 GB in bf16, which matters when the fit is within a few gigabytes.
• "Would it fit on one H200?" 141 GB of weights into 141 GB of memory leaves nothing for the cache, so no in bf16; fp8 weights at 71 GB leave 56 GB usable, which is a real deployment.
• "What about the 405B?" Same formula: 405e9 × 2 = 810 GB in bf16, more than a full 8 × H100 node's 640 GB, so it needs fp8 (405 GB) on one node or bf16 across two.
• "How does the number change for a MoE model?" Memory follows total parameters, so DeepSeek-V3 at 671B needs 671 GB in fp8 even though only 37B are active per token.

Common mistakes

• Quoting "70 GB" for a 70B model, which is the fp8 number, without saying so. The interviewer cannot tell whether the candidate knows the precision or dropped the factor of two.
• Treating the weight number as the serving footprint and concluding two H100s serve a bf16 70B comfortably.
• Using GiB against a card quoted in GB, which makes a 141 GB model look like it fits in 2 × 80 GB with more room than it has.
• Applying the active-parameter count to a mixture-of-experts model's memory. Active parameters set compute and bandwidth per token, never the memory the weights occupy.

Key takeaways

• Weights = parameters × bytes per parameter: 141 GB bf16, 71 GB fp8/int8, 35 GB int4 for Llama 3.1 70B.
• Say the precision with every size, and use decimal GB to match the card.
• Weights are the floor; the KV cache (320 KB per token here) and about 10% overhead sit on top.
• One H100 holds the fp8 weights and almost no cache; the useful configurations are TP4 or TP8 in bf16, or fp8 on fewer cards.