KV Cache Sizing
Derive cache payload from the stored attention state, then account for mixed sequence lengths, block allocation and per-device limits. Worked examples distinguish exact bytes from deployable capacity.
TL;DR: For uniform full-attention layers with equal key and value dimensions, KV payload per retained position is
2 × layers × KV heads × head dimension × cache bytes per element. Llama 3.1 70B with a BF16 cache needs 320 KiB per position: 2.5 GiB at 8,192 positions and 40 GiB at 131,072. Allocation, sharing and per-device placement turn that payload into a capacity plan.
Count the state before counting users
A model can fit in memory and still run out of room as conversations grow. Its weights stay roughly fixed, while its KV Cache retains attention state for the tokens it has consumed. A capacity estimate therefore needs both the model and the current sequence lengths.
Start with one retained token position at one layer. Each key-value head contributes a key vector and a value vector. If each vector has d elements, h KV heads store 2 × h × d values. Multiply by the number of layers, then by the bytes used to store each value.
L = layers; h = KV heads; d = elements per head; s = cache bytes per element
payload per position = 2 × L × h × d × s
The leading two counts K and V. Query vectors are not another persistent cache term. Grouped-query attention lets several query heads use one KV head; substituting the query-head count overstates the cache.
Read the architecture and the runtime separately
NVIDIA's Llama 3.1 70B configuration has 80 layers, 8 KV heads and 64 query heads. Its hidden width is 8,192, giving a head dimension of 128 for this architecture. With an explicitly assumed BF16 cache:
2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KiB per position
320 KiB × 8,192 positions / 1,048,576 KiB per GiB = 2.5 GiB
320 KiB × 131,072 positions / 1,048,576 = 40 GiB
BF16 is the cache format in this calculation. A checkpoint with FP8 weights can still use another cache format. Read the engine's resolved cache configuration and allocation report rather than inferring it from the model name.
Also check for an explicit head dimension. Qwen3-32B's config specifies head_dim: 128, even though hidden_size / num_attention_heads = 5120 / 64 = 80. Blindly using that division gives the wrong answer.
| Configuration, BF16 cache | Per position | At 8,192 positions |
|---|---|---|
| Llama 3.1 70B: L=80, h=8, d=128 | 320 KiB | 2.5 GiB |
| Qwen3-32B: L=64, h=8, d=128 | 256 KiB | 2 GiB |
These are payload calculations, not measurements of a running engine. The bytes follow attention state, not total parameter count.
Add lengths, not just users
For unshared full-attention caches, sum the retained lengths and multiply by the per-position payload. A chat with 8,192 retained positions costs less than one with 32,768, even though both count as one user.
Here is a hypothetical workload on the Llama configuration above. All lengths are retained positions, including prompt state; there is no shared prefix or eviction.
Sanity check: a 32,768-position sequence is four times an 8,192-position sequence. Two long sequences therefore cost twice as much as four short ones.
PagedAttention rounds allocations to block boundaries. For unshared sequences with lengths Tᵢ and block size b, count sum(ceil(Tᵢ / b) × b) allocated positions. Round each sequence before summing. With shared prefixes, count unique physical blocks instead; summing logical lengths would count shared data repeatedly.
Check each device's budget
A rank is one participating worker, usually attached to a GPU. An aggregate fleet budget can hide a full rank. For each rank, subtract actual weights, activation and workspace peaks, runtime allocations and deliberate headroom from available memory. Compare its cache allocation against that remainder.
For a hypothetical tensor-parallel layout that evenly shards eight KV heads across eight ranks without replication, the 40 GiB sequence payload becomes 5 GiB per rank. Ten such sequences need 50 GiB on every rank. If seven ranks have 52 GiB available but one has 47 GiB, the ten-sequence plan fails. The limiting rank permits floor(47 / 5) = 9 current allocations before further headroom.
Do not divide by the GPU count unless the engine actually shards the cache that way. Replication, pipeline placement, mixed layer types and extra states can change the calculation. Likewise, a fixed 10% reserve is a planning assumption, not a measured runtime requirement.
Know when the formula changes
| Architecture or storage change | What to count |
|---|---|
| Unequal K and V dimensions | Add their dimensions instead of multiplying one dimension by two |
| Sliding-window or hybrid layers | Sum each layer's retained state and its own retention rule |
| Compressed latent attention | Count the stored latent representation rather than expanded heads |
| Quantized cache | Count stored value bytes plus scales, padding and metadata |
For example, DeepSeek-V3's configuration has 61 main layers, a 512-wide KV latent and a 64-wide positional key. An implementation retaining those in BF16 has a main-layer payload of 61 × (512 + 64) × 2 = 70,272 bytes = 68.625 KiB per position. This assumes compressed storage; extra prediction state, replication or an expanded backend layout requires additional accounting.
An FP8 cache halves the raw value bytes relative to BF16. It does not promise exactly twice the usable concurrency: scale storage, allocation granularity and other limits remain. vLLM's quantized-cache documentation describes the precision and scaling requirements. Test long-context output quality and backend support before choosing it.
Check your understanding
All six conversations in the chart grow by 4,096 retained positions. How much extra BF16 payload is needed? Each needs 4,096 × 320 KiB = 1.25 GiB; six need 7.5 GiB, bringing the total to 37.5 GiB. A 35 GiB payload budget held the original workload but cannot hold this growth.
Key takeaways
- Count two vectors, KV heads, layers, head dimension and cache precision explicitly.
- 320 KiB per position becomes 40 GiB at 131,072 positions for the stated Llama configuration.
- Sum actual lengths, account for block allocation and sharing, then check every rank.
- Capacity is only one constraint. A fitting workload still needs to meet its latency and quality targets.