TL;DR: The KV cache retains attention keys and values for tokens the model has already processed. An ordinary decode step reuses that state and adds the position it consumes. For Llama 3.1 70B with a BF16 cache, the payload is 320 KiB per retained position, so longer contexts and more active sequences need more memory. Capacity still depends on the actual per-device allocation and growth policy.
How to approach it
First distinguish a consumed token from the token just emitted. Trace two decode steps, then derive the payload from layers, KV heads, head dimension and precision. Compare it with a cache budget after weights and runtime allocations. Close by reserving room for continued generation and checking latency separately.
A strong answer
A response keeps getting longer while the model weights stay unchanged. What is taking the extra memory? Attention state for more positions. The KV Cache holds the keys and values used when later positions attend to earlier context.
For an ordinary causal decoder (a model in which each token can only look at earlier tokens, as in chat models), future tokens do not change the already computed states of past tokens. Each layer can retain those K/V tensors and reuse them. Attention for the newly consumed position uses past state and its own current K/V; it does not attend to future positions. Hugging Face's cache explanation shows this update and the subsequent selection of the next token.
The timing distinction matters. Consuming prompt tokens A and B produces scores from which C is selected. The cache covers A and B. The next iteration consumes C, computes its state and emits D. The last emitted token may never be consumed if generation stops. These are symbolic IDs, not a tokenizer example. Speculative decoding can process candidates in larger groups; the picture describes ordinary one-token decode.
Caching avoids repeating old token projections and layer work. It does not make total generation cost linear in output length: full attention still reads an increasing history on later steps. Prefill vs Decode explains why processing a prompt and extending a sequence have different execution shapes.
Count the vectors before the bytes. A retained position contributes K and V for each KV head in each layer. With uniform full-attention layers and equal key/value dimensions:
KV bytes per position = 2 × layers × KV heads × head dimension × bytes/value
Llama 3.1 70B example, assumed BF16 cache:
= 2 × 80 × 8 × 128 × 2
= 327,680 bytes = 320 KiB
sanity: 64 query heads / 8 KV heads = 8 query heads sharing each KV head.
Counting query heads would overstate this cache by 8×.
The dimensions come from NVIDIA's published Llama 3.1 70B configuration: head dimension is 8,192 / 64 = 128. That artifact's weight format does not determine our assumed cache precision. For architectures with different layer types or compressed state, derive their actual stored tensors instead of applying this formula unchanged.
| Retained positions | BF16 payload per sequence |
|---|---|
| 4,096 | 1.25 GiB |
| 32,768 | 10 GiB |
| 131,072 | 40 GiB |
These are binary units: KiB = 1,024 bytes and GiB = 1,024³ bytes. Each row is positions × 327,680 bytes. Eight times the retained context produces eight times the payload; the configured maximum context is not a promise that a given deployment has that much free memory.
Admission needs room for growth. Use a hypothetical 40 GiB budget already available for KV payload after all other allocations. Ignore page rounding and sharing for this exercise. With 4,096 retained positions per request, the instantaneous payload bound is 40 / 1.25 = 32 requests. If every request may consume another 1,024 positions before finishing, reserve for 5,120:
reserved payload/request = 5,120 × 320 KiB = 1.5625 GiB
growth-reserved request bound = floor(40 / 1.5625) = 25
25 reservations use 39.0625 GiB; 26 would use 40.625 GiB.
sanity: 32 current payloads fit, but 32 growth reservations do not.
That is a payload exercise, not a GPU concurrency benchmark. A real scheduler accounts for allocated blocks, shared prefixes, runtime workspace and its preemption policy. On a distributed replica, each device must fit its own shard; adding all GPUs' free memory can hide an overloaded rank. KV Cache Sizing develops that boundary.
When cache space becomes scarce, compare free blocks, waiting requests and preemptions with the gaps between streamed tokens. Low SM utilization (the share of time the GPU's streaming multiprocessors are busy) alone cannot tell you that cache capacity is the cause. A preempted request may later need recomputation or a state transfer, which adds work. Finishing or cancelling releases the request's references, but a prefix-cache policy may retain reusable blocks. An ongoing chat session therefore does not necessarily keep its previous state on the GPU between turns.
Choose the smallest state representation that meets model quality and execution requirements. GQA reduces KV heads; supported cache quantization reduces bytes per value; compatible prefix sharing reduces duplicate physical state. Paging reduces reservation waste rather than bytes in a useful position. vLLM's quantized-cache documentation describes backend and scaling considerations. A one-byte cache halves the raw BF16 payload here, before metadata, but does not establish a latency or quality gain. If memory has headroom and another resource limits service, changing cache precision may add complexity without helping the objective.
What interviewers probe next
- "Does the last output token already occupy KV space?" Not in the ordinary loop until it becomes input to another model iteration.
- "What if context doubles?" Full-attention payload per sequence doubles. That alone does not predict a 2× latency change.
- "Does sliding-window attention keep growing forever?" A bounded retained window can cap that layer's cache. Sum each layer type's actual storage.
- "Why not admit all 32 requests in the example?" They fit now, but the stated completion reservation permits only 25; dynamic admission would need an explicit waiting or preemption policy.
Common mistakes
- Using query-head count where the cache stores KV heads.
- Treating GB and GiB as interchangeable during capacity arithmetic.
- Counting generated tokens while omitting the prompt's retained state.
- Dividing total GPU memory by cache size without weights, workspace or shard limits.
- Promising a speedup directly from bytes saved.
Key takeaways
- The ordinary loop caches consumed positions; a newly selected token is processed next.
- This 70B BF16 example stores 320 KiB per position and 1.25 GiB at 4,096 positions.
- A 40 GiB payload budget fits 32 current payloads but only 25 reservations for 5,120 positions.
- Cache size, physical allocation and latency are different measurements.