KV Cache Sizing
The KV cache is the memory that decides how many users a serving replica can hold and how long their context can be. Its size per token comes from four numbers in the model's config file (layers, KV heads, head dimension, bytes per element) and one formula; multiplied by context and concurrency it is the number every capacity plan is built on. This page derives it, works it for four models including an MLA one, and shows the two places candidates get it wrong by a factor of eight.
TL;DR: KV bytes per token = 2 × n_layers × n_kv_heads × d_head × bytes_per_element. The 2 is K and V; n_kv_heads is the key-value head count (8 on Llama 3.1, not the 64 query heads); d_head is usually 128; bytes is 2 for bf16 and 1 for fp8. Llama 3.1 70B: 2 × 80 × 8 × 128 × 2 = 320 KB per token, so 42 GB per 128k-token sequence and 2.6 GB per 8k one. For MLA models (DeepSeek-V3): n_layers × (latent + rope) × bytes = 61 × 576 × 2 ≈ 70 KB. Total = per token × context × concurrent sequences, and the free budget is the card minus weights minus about 10%.
Why every token costs memory
During decode, each new token attends over every previous token, and attention needs each previous token's key and value vectors at every layer. Recomputing them every step would make decode quadratic; caching them makes it linear at the price of storing, for every token in every live sequence, one K and one V vector per layer per KV head. That storage is the KV cache (the mechanism). Its size is architecture arithmetic, and it is worth doing slowly once.
per token, per layer:
one K vector and one V vector per KV head, each d_head elements
= 2 × n_kv_heads × d_head elements
per token, whole model:
× n_layers
in bytes:
× bytes_per_element (2 for bf16/fp16, 1 for fp8)
KV bytes per token = 2 × n_layers × n_kv_heads × d_head × bytes
The four inputs come straight from the model's config.json: num_hidden_layers, num_key_value_heads, hidden_size ÷ num_attention_heads (the head dimension, usually 128), and the dtype you serve in.
Worked for four models
| Model | layers | KV heads | d_head | per token (bf16) | 8k context, one sequence | 128k context, one sequence |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 128 | 2 × 32 × 8 × 128 × 2 = 131,072 B = 128 KB | 1.07 GB | 17 GB |
| Llama 3.1 70B | 80 | 8 | 128 | 2 × 80 × 8 × 128 × 2 = 327,680 B = 320 KiB | 2.6 GB | 42 GB |
| Qwen3 32B | 64 | 8 | 128 | 2 × 64 × 8 × 128 × 2 = 262,144 B = 256 KB | 2.1 GB | 34 GB |
| Mixtral 8x7B | 32 | 8 | 128 | 128 KB (same shape as Llama 8B despite 47B params) | 1.07 GB | 17 GB |
Two things to notice in the table. Mixtral 8x7B has 47 billion parameters and the KV cache of an 8B model, because the cache depends on layers and KV heads, not on parameter count; the experts add parameters without adding attention state. And the 70B and 8B models have the same KV heads (8) and head dimension; the 2.5x difference in cache is purely the layer count, 80 versus 32.
Now the MLA case, which the table's formula does not cover. DeepSeek-V2, V3 and R1 store a compressed latent per token per layer instead of full keys and values, plus a small decoupled RoPE key:
MLA: KV bytes per token = n_layers × (d_latent + d_rope) × bytes
DeepSeek-V3: 61 × (512 + 64) × 2 B = 61 × 576 × 2 = 70,272 B ≈ 70 KB
sanity: a 671B-parameter model with a per-token cache a fifth of Llama 70B's.
Had V3 used GQA with 8 heads of 128 at 61 layers it would be 250 KB; the latent is what makes it small.
Put the five side by side and the point of the whole page is visible in one look: the bars track layers and KV heads, and they ignore parameter count completely.
Mixtral and Llama 8B are the same bar with a 6x difference in parameters, and DeepSeek-V3 is the shortest bar on the chart with the largest model on it.
From per token to a capacity plan
Total cache is per token times context times concurrent sequences, and the budget it must fit in is the card's memory minus the weights minus a reserve.
serve Llama 3.1 70B in bf16 on 8 × H100, 64 users at 32k context: does it fit?
capacity: 8 × 80 GB = 640 GB; usable at 90%: 576 GB
weights: 141 GB → free for KV: 576 − 141 = 435 GB
per sequence at 32k: 320 KB × 32,768 = 10.5 GB
64 sequences: 64 × 10.5 = 671 GB
verdict: 671 GB > 435 GB, it does not fit in bf16
max users at 32k in bf16: 435 ÷ 10.5 ≈ 41
with an fp8 KV cache (bytes 2 → 1): per sequence 5.2 GB, max users ≈ 82, the 64 fit
with fp8 weights too (71 GB): free 505 GB, ≈ 96 users at 32k
That is the calculation behind the sample answer on the home page and behind most "will it serve" questions. The KV cache calculator runs it for any model and card.
The two mistakes that cost a factor of eight
Using the query-head count. Llama 3.1 70B has 64 attention heads and 8 KV heads; grouped-query attention shares each KV head across 8 query heads. Plugging 64 into the formula gives 2.6 MB per token and a capacity plan eight times too pessimistic. The interviewer will ask "how many KV heads?" and is checking for exactly this.
Forgetting the factor of 2. Each token stores both a key and a value. Dropping it halves the estimate. Say "two, for K and V" out loud as the first factor.
A third, smaller one: mixing up decimal and binary units. 320 KB per token is 327,680 bytes; at 32,768 tokens that is 10.7e9 bytes, which is 10.7 GB decimal or 10.0 GiB. Vendors quote card memory in decimal GB, so stay decimal throughout and the comparisons are honest.
Working it in the room
"How big is the KV cache for Llama 70B at 128k?" wants the formula stated with its variables named, the four numbers pulled from memory or asked for, the multiplication done in two steps (per token, then per sequence), and the sanity check against a card. The follow-up held back is "and how would you cut it?", in the order that matters: fp8 KV (halves it), a model with fewer KV heads or MLA (an architecture choice, not a serving one), paged allocation to stop over-reserving, and prefix sharing so common prompts are stored once. The answer that sounds right and fails is quoting a number without the KV-head count: the interviewer cannot tell whether you got lucky.
What to remember
- KV per token = 2 × layers × KV heads × head dim × bytes. Say "two, for K and V" first.
- Llama 3.1 70B: 320 KB per token (bf16); 8B: 128 KB; Qwen3 32B: 256 KB. DeepSeek-V3 with MLA: about 70 KB despite 671B parameters.
- Per sequence: 2.6 GB at 8k and 42 GB at 128k for the 70B. Budget = usable card memory (90%) minus weights.
- 8 × H100 serving 70B bf16 holds about 41 users at 32k in bf16 and about 82 with an fp8 cache.
- The eight-times mistake is the query-head count; the two-times mistake is dropping K-and-V.