Core

Attention Variants: MHA, GQA, MQA and MLA

The KV cache scales with the number of key-value heads, and the four attention variants differ exactly there: multi-head keeps one KV head per query head, multi-query keeps one for all, grouped-query shares one across a group, and multi-head latent attention caches a compressed latent instead of keys and values at all. For Llama 3.1 70B that is the difference between 2.6 MB and 320 KB per token; for DeepSeek-V3 it is about 70 KB. The variant a model was trained with is a serving decision made before the first GPU was bought.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

The KV Cache KV Cache Sizing Memory-Bound vs Compute-Bound Kernels FlashAttention Internals Bandwidth-Bound Decode Throughput PagedAttention

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureThe KV cache is what actually decides how many users fit14 min→Inference EngineeringThe model author already made your biggest capacity decision14 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingCompare the KV cache footprint of multi-head, grouped-query and multi-head latent attention with numbers.→LLM Inference & ServingWhat changes when you serve DeepSeek-V3 compared with a dense 70B model?→LLM Inference & ServingWhat is the KV cache, and why does it keep growing while a request is being served?→Napkin Math, Cost & CapacityHow big is the KV cache for Llama 3.1 70B at a 128k context?→Open-Weights Models & Serving EnginesThe model advertises a million tokens of context. What actually breaks when you serve it?→LLM Inference & ServingYou need to serve 128k-token contexts. What breaks first, and what do you change?→