KV cache & memory-fit calculator
Estimate KV cache per token, sequence and batch for a model or custom architecture. Compare weights, cache and an adjustable reserve against total device memory, with formulas and placement limitations.
Model
70.6B params · 80 layers · 8 KV heads × 128 head dim
Workload
Hardware
This adds device capacities before checking weights and KV demand. Each GPU must still fit its assigned tensors in a supported parallel layout. Quantization metadata and engine allocations can change the result; throughput and latency require a workload measurement. GB and KB here use decimal units.
- KV per token
- 2 × 80 × 8 × 128 × 2 B = 328 KB
- KV total
- 327,680 B × 8,192 × 32 = 85.9 GB
- Weights
- 70.60B × 2 B = 141 GB
- KV budget / deficit
- 80.0 GB − weights − 8.0 GB reserve = -69.2 GB
Use the result to make a decision
Start with the model geometry and storage formats, calculate retained KV bytes, then subtract weights and runtime reserve from available memory. Check the budget on every GPU in the proposed layout. This gives a memory limit; a workload benchmark must establish throughput and latency.
Questions people ask
For full-attention MHA/GQA: 2 × layers × KV heads × head dimension × bytes per element. Llama 3.1 70B with BF16 KV uses 327,680 bytes per token: 320 KiB or 327.68 decimal KB. At 8,192 retained tokens that is 2,684,354,560 bytes, about 2.68 GB per sequence. Sliding-window, hybrid and MLA layouts need their own accounting.
Multi-head latent attention caches one compressed 512-wide latent and a 64-wide RoPE key per token per layer instead of full keys and values for every head. At 61 layers and bf16 that is about 70 KB per token, roughly a fifth of Llama 70B, despite a much larger model.
The initial 10% is an editable teaching assumption, not a measured requirement or an engine default. Runtime buffers, activations and captured graphs vary with the engine and workload. Measure non-weight, non-KV memory at the intended peak; an engine utilization fraction can include weights and cache and is not automatically an overhead percentage.
Changing two-byte BF16 cache elements to one-byte FP8 halves the raw tensor payload for the same layout. Scales, padding and layers left at another dtype affect actual allocation. Verify model/backend support and evaluate quality, latency and throughput; the payload ratio does not guarantee twice the serving capacity.