The KV Cache
The KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.
TL;DR: Every decode step needs the keys and values of every earlier token, so the engine keeps them in HBM instead of recomputing them, at 2 × layers × KV heads × head dimension × bytes per token. That is 320 KB per token for Llama 3.1 70B in bf16, so a single 128k-token context costs about 43 GB and the cache, not the weights, is what runs out first under load.
What is cached and why
Attention for a new token computes a query vector, then scores it against the key of every previous token and mixes their values. The keys and values of a token do not change once it is in the context: the same token at the same position produces the same K and V in every later step. Without a cache, decode step n recomputes K and V for all n prior tokens, and the total work over a response of length n grows with n squared. With the cache, each token's K and V are computed once, during prefill or on the step that emitted it, and read back on every later step.
The trade is compute for memory. Reads are cheap on paper; the cost is that the cache lives in HBM for the life of the request, alongside the weights, and it grows one entry per layer per token for as long as the sequence runs.
The per-token formula
For standard multi-head or grouped-query attention, each layer stores one key and one value vector per KV head:
bytes per token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
The leading 2 is K and V. Worked for the presets in the site's model table, all in bf16 (2 bytes):
| Model | Layers | KV heads | Head dim | Per token | 8k context | 128k context |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 128 | 128 KB | 1.1 GB | 17.2 GB |
| Llama 3.1 70B | 80 | 8 | 128 | 320 KB | 2.7 GB | 43 GB |
| Llama 3.1 405B | 126 | 8 | 128 | 504 KB | 4.1 GB | 66 GB |
| Qwen3 235B-A22B | 94 | 4 | 128 | 188 KB | 1.6 GB | not supported |
| DeepSeek-V3 (MLA) | 61 | latent 512 + rope 64 | about 70 KB | 0.6 GB | 9.2 GB |
Two things jump out. The 405B model has a smaller cache per parameter than the 8B because both use 8 KV heads; the cache scales with layers, not with model width. And DeepSeek-V3 caches a single 576-wide latent per layer with no factor of two, which is why a 671B-parameter model has a cache one fifth the size of a 70B dense model's.
Why it decides concurrency
The cache budget is whatever HBM is left after weights and working memory. Llama 3.1 70B in bf16 on two H100s under tensor parallelism has 160 GB total, about 141 GB of it weights, leaving under 15 GB after activations and CUDA context. At 320 KB per token that is roughly 45k cached tokens across all requests: five concurrent sequences at 8k, or a single request at 45k. Move the same model to four H100s and the budget becomes about 170 GB, over 500k tokens. The weights did not change; the batch you can serve went up ten times.
Engines log this budget at startup. vLLM prints a line of the form "GPU KV cache size: N tokens" followed by "Maximum concurrency for M tokens per request", and that pair is the first thing an operator reads on a new deployment. If N divided by your typical context is smaller than your target batch, you will preempt requests under load, and the num_preemptions counter climbing is the confirmation.
The cache as a bandwidth cost
Memory is the visible cost. The hidden one is that every decode step reads the entire cache of every sequence in the batch. At 8k context, one Llama 70B sequence reads about 2.7 GB of cache per step. A batch of 64 reads about 170 GB per step, more than the 141 GB of weights. Past that point the step time is dominated by cache traffic, throughput per added sequence flattens, and long-context batches on the same hardware run measurably slower per token than short-context ones. This is the mechanism behind "TPOT got worse when we raised the context limit", and it is why GQA and MLA exist.
What interviewers are listening for
They want the formula stated from memory and worked for one named model, with the assumptions (dtype, layers, KV heads) said out loud. Then they push: "your KV budget is 40 GB, context is 32k, what batch do you get" (about four sequences for the 70B). The follow-up held in reserve is "what changes for a mixture-of-experts model". The answer that sounds right and fails is "the cache shrinks because only some experts are active". Experts sit in the MLP; attention runs every layer for every token, so the cache is set by the attention configuration alone. Qwen3 235B-A22B caches 188 KB per token despite activating 22B parameters.
Common misconceptions
The cache is not the weights. Weights are fixed per model; the cache scales with batch times context and is what fails first.
Quantizing weights does not shrink the cache. The cache has its own dtype (fp8 KV halves it) and its own switch.
A longer supported context does not cost anything until someone uses it. A 128k limit with 2k-token traffic uses the same memory as a 2k limit; the scheduler reserves by actual length when paged.
Key takeaways
- Bytes per token = 2 × layers × KV heads × head dim × bytes; 128 KB for Llama 3.1 8B, 320 KB for 70B, about 70 KB for DeepSeek-V3 in bf16.
- Total cache = per token × context × batch, and it competes with weights for HBM; leftover HBM decides batch.
- Every decode step re-reads every sequence's cache, so long contexts slow every token, not just the first.
- MoE does not shrink the cache; attention heads and layers set it.
- Read the engine's KV budget line at startup and watch preemptions; both tell you when the cache is the bottleneck.