Core

Attention Variants: MHA, GQA, MQA and MLA

Compare MHA, GQA, MQA and compressed MLA by the state each retains. Derive cache payload in consistent units, distinguish storage capacity from serving throughput, and account for hybrid layers and rank replication.

This concept is included with a free account. Sign in to continue reading.

free account · no card needed

RELATED CONCEPTS

The KV Cache KV Cache Sizing Memory-Bound vs Compute-Bound Kernels FlashAttention Internals Bandwidth-Bound Decode Throughput PagedAttention

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureTurn retained context into a cache capacity limit14 min→Inference EngineeringAttention architecture determines which state the serving engine must retain29 min→Kernels and the MachineIO-aware attention is a memory-traffic algorithm, not a faster matmul29 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingCompare the KV cache footprint of multi-head, grouped-query and multi-head latent attention with numbers.→LLM Inference & ServingWhat changes when you serve DeepSeek-V3 compared with a dense 70B model?→LLM Inference & ServingWhat is the KV cache, and why does it keep growing while a request is being served?→Napkin Math, Cost & CapacityHow big is the KV cache for Llama 3.1 70B at a 128k context?→Open-Weights Models & Serving EnginesYour deployment is limited by KV cache rather than by compute. What are the options?→Open-Weights Models & Serving EnginesThe model advertises a million tokens of context. What actually breaks when you serve it?→