TL;DR: With a BF16 cache and 131,072 retained positions, Llama 3.1 70B stores
2 × 80 × 8 × 128 × 2 × 131,072 = 42,949,672,960payload bytes: 40 GiB, or about 42.95 GB. Use its eight KV heads, not its 64 query heads. Physical allocation and placement still need checking before turning that number into a user limit.
How to approach it
Clarify what 128k means and which precision the cache uses. State the KV Cache Sizing formula with each variable named, calculate bytes per position, then multiply by retained positions. Finish with the distinction between this one sequence's payload and what each GPU must allocate.
A strong answer
Imagine a team has already loaded the model and now wants to enable long conversations. The weight footprint did not grow, but each conversation brings more attention history. That history can consume tens of GiB before the answer has finished.
The KV Cache holds keys and values for consumed token positions. For this model's uniform full-attention layers, each position stores two vectors per KV head per layer. Grouped-query attention shares those KV heads across query heads; the cache does not store a separate copy for every query head.
NVIDIA's published Llama 3.1 70B configuration supplies the architecture dimensions. BF16 cache storage is our explicit runtime assumption, independent of that checkpoint's weight format.
| Input | Value |
|---|---|
| Layers | 80 |
| KV heads per layer | 8 |
| Elements per K or V head | 128 |
| Bytes per cached value, BF16 | 2 |
| Retained positions, our definition of 128k | 131,072 |
per position = 2 (K and V) × 80 × 8 × 128 × 2 bytes
= 327,680 bytes
= 320 KiB
one sequence = 327,680 × 131,072
= 42,949,672,960 bytes
= 40 GiB exactly
≈ 42.95 GB decimal
Sanity check: 131,072 is sixteen times 8,192. The same configuration at 8,192 positions uses 2.5 GiB, and 16 × 2.5 = 40 GiB. The 128k count includes retained prompt and generated-token state; it is not an extra output allowance on top of a full prompt. The configured context limit and any needed output space are separate admission constraints.
Query heads read shared state; they do not each require a duplicate stored pair. The boxes show the eight stored head groups, not eight GPUs until the stated sharding assumption is applied.
At one byte per cached value, the corresponding FP8 value payload is 20 GiB. Scales and other storage are additional. Changing only the weights to FP8 leaves this BF16 cache calculation unchanged. vLLM's quantized-cache documentation also makes clear that scaling and backend support matter; measure task quality rather than treating reduced precision as a free capacity switch.
To turn the model-wide payload into a capacity estimate, suppose each of eight ranks has 48 GiB left for KV after other allocations and headroom. Also assume the engine places one of the eight KV heads on each rank, with no duplication. Each full-length sequence then needs 40 / 8 = 5 GiB on every rank. Nine fit within each 48 GiB budget; ten would need 50 GiB. Dividing the total is valid here because the storage layout and equal rank budgets were stated explicitly.
That is a hypothetical budget, not an benchmark. Confirm the cache layout and the most constrained rank. If one rank has only 43 GiB, the ceiling falls to eight even if the fleet's total free memory looks sufficient. Available cache blocks, preemptions and per-rank allocation reports are more useful evidence than a fixed reserve percentage.
PagedAttention changes how much unused space is reserved, not the value payload at a fully retained length. Compatible shared prefixes can reduce unique physical storage; block padding or replicated state can increase it. For an unshared sequence at 131,072 positions, 16-position blocks have no partial-tail waste because the length is an exact multiple of 16.
I would consider a lower-precision cache if cache capacity is the binding constraint and quality checks pass. I would reverse that choice if latency or quality becomes the limiting factor. A cache that fits does not prove a 128k prompt can be processed or decoded within the product's time budget.
What interviewers probe next
- “What happens if you use 64 heads?” The estimate rises eightfold to 320 GiB. Those are query heads, so they do not belong in this model's KV-head term.
- “What if 128k means 128,000?” Compute it explicitly:
327,680 × 128,000 = 41,943,040,000 bytes = 39.0625 GiB. State the interpretation rather than silently rounding. - “Can I divide the result by any GPU count?” Only if the actual layout shards the retained state evenly without replication. Check the backend, tensor-parallel degree and layer placement.
- “Does the same formula apply to MLA?” Not to a compressed latent cache. Count its stored representation; KV Cache Sizing shows the alternative calculation.
Common mistakes
- Saying 320 KB when the exact result is 320 KiB, or comparing unconverted units.
- Reading cache precision from the weight-format label.
- Reserving the whole supported context for every short request without checking allocation behavior.
- Treating payload capacity as a throughput or latency guarantee.
Key takeaways
- The stated BF16 configuration needs 320 KiB per retained position and 40 GiB at 131,072 positions.
- Eight KV heads matter; substituting 64 query heads creates an eightfold error.
- An evenly sharded eight-rank layout needs 5 GiB per rank per full-length sequence before extras.
- Derive the payload, then verify precision, allocation, placement and the workload's latency target.