KV cache & memory-fit calculator

Per token: 2 × layers × KV heads × head dimension × bytes per element. Multiply by context length for one sequence and by the number of concurrent sequences for the batch. Llama 3.1 70B in bf16 is 2 × 80 × 8 × 128 × 2 = 320 KB per token, so an 8k-token sequence holds about 2.6 GB of cache.

Multi-head latent attention caches one compressed 512-wide latent and a 64-wide RoPE key per token per layer instead of full keys and values for every head. At 61 layers and bf16 that is about 70 KB per token, roughly a fifth of Llama 70B, despite a much larger model.

Around 10% of the card for the CUDA context, allocator fragmentation and activations, more if you run large batches or long prefills. Serving engines expose this as a memory-utilization fraction; vLLM defaults to using 90% of the card.

Yes for capacity, and it is common in production for long-context serving. The cost is a small accuracy loss that you should measure on your own evals rather than assume, and it does not help prefill compute at all.