TL;DR: MHA, GQA and MQA change how many query heads share each stored key/value head. MLA stores a compressed representation shared across heads plus positional state. Count the actual stored tensors, then apply precision and layer count. A smaller cache does not establish that a larger model fits, runs faster or has equal quality.
How to approach it
First separate a controlled change in head sharing from a comparison between different models. Draw which query heads use which stored state. Derive per-position bytes and one context-length example, then check the engine's physical cache representation before turning payload into capacity.
A strong answer
Two checkpoints can have very different cache sizes even when their weight files look similar. Check how each model stores attention state before using its free memory to estimate request capacity. The KV Cache stores the state needed by later positions; it is not a copy of all attention outputs.
Attention Variants: MHA, GQA, MQA and MLA explains the mechanisms. Multi-head attention (MHA) gives each query head its own K/V head. Grouped-query attention (GQA) shares one K/V head among a group of query heads. Multi-query attention (MQA) shares one across all query heads in a layer. Multi-head latent attention (MLA), covered below, stores a compressed representation instead. The GQA paper studies the speed/quality tradeoff and an uptraining recipe; it does not make head sharing an inference flag that preserves any checkpoint unchanged.
The four-head sketch isolates ownership. For the numeric comparison below, hold 80 layers, 64 query heads, head width 128 and BF16 storage fixed, and vary KV heads. MHA and MQA rows are hypothetical variants of that geometry; the eight-head GQA geometry matches Llama 3.1 70B. It is not a comparison of equally trained models.
payload/position = 2 × layers × KV heads × width × bytes/value
MHA: 2 × 80 × 64 × 128 × 2 = 2,621,440 B = 2,560 KiB
GQA: 2 × 80 × 8 × 128 × 2 = 327,680 B = 320 KiB
MQA: 2 × 80 × 1 × 128 × 2 = 40,960 B = 40 KiB
sanity: changing only 64 KV heads to 8 divides payload by 8.
At 32,768 retained positions those payloads are 80 GiB, 10 GiB and 1.25 GiB respectively. All use the same binary units and omit allocator overhead, growth and replication. KV Cache Sizing explains why each rank's real budget still matters.
MLA changes the representation. In the compressed decode form, a latent carries information used for both keys and values, alongside a decoupled rotary-position key. The DeepSeek-V3 report describes matrix absorption that avoids materializing the complete per-head K/V cache during decoding. Thus the stored latent is not multiplied by two merely because it serves both roles.
For the V3 geometry of 61 layers, latent width 512 and positional key width 64, assume BF16 for both stored components:
MLA payload/position = 61 × (512 + 64) × 2
= 70,272 bytes = 68.625 KiB
at 32,768 positions = 2,302,672,896 bytes = 2.14453125 GiB
sanity: 70,272 / 327,680 ≈ 0.2145 of the GQA example's payload.
This compares different architectures with different layer counts, not equal-size models. It says nothing by itself about total weight memory or how many GPUs DeepSeek-V3 requires. Quantization, kernel layout or a cache implementation that stores expanded tensors can change the physical result; inspect the actual allocation.
| Stored state | Question for the implementation |
|---|---|
| Separate per-head K/V | How many KV heads are stored on each rank? |
| Shared K/V heads | Are any heads replicated by the parallel layout? |
| Compressed latent plus positional key | Does this backend retain that compressed form? |
| Quantized state | What scales, padding and additional tensors exist? |
Do not assign universal quality labels to these rows. Architecture and training determine the model's behavior, and a serving team usually chooses a checkpoint rather than freely replacing its attention layout. Likewise, MLA's smaller retained payload does not prove a faster kernel. Matrix absorption changes the executed arithmetic, and parallelism, context length and hardware determine the bottleneck.
Choose a supported model/backend combination that passes the task's quality requirements, then measure per-rank cache allocation, p99 token gaps and admitted work. If long-context cache capacity is binding, the representation deserves attention. If weights, compute or communication dominate, a smaller per-position number may not improve service. Cache quantization is a separate experiment, with backend support and quality checks of its own.
What interviewers probe next
- "Why does the MLA formula lack a factor of two?" One stored latent supplies both K and V content; the separate positional key is counted once too.
- "Can I change num_key_value_heads in config.json?" Not safely as a format edit. The checkpoint's trained projections and shapes must support the architecture; conversion requires an appropriate training procedure.
- "Does FP8 halve total cache memory?" It halves the raw value bytes if both components use one byte, but scales and allocation overhead remain.
- "Which one has the best quality?" Compare trained models on the required tasks. A byte-count derivation cannot answer that.
Common mistakes
- Comparing different architectures as if only one field changed.
- Counting query heads instead of stored KV heads.
- Applying the K-plus-V factor twice to a shared latent.
- Inferring device capacity or quality from cache payload alone.
Key takeaways
- The controlled 80-layer examples store 2,560, 320 and 40 KiB per position.
- The stated V3 compressed BF16 form stores 68.625 KiB per position.
- At 32,768 positions, that MLA payload is about 2.145 GiB, excluding other memory.
- Verify physical representation, parallel layout and service behavior before choosing capacity.