Multi-Head Latent Attention and Sparse Indexers

TL;DR: Latent attention compresses retained keys and values. Sparse indexing selects a smaller set for expensive attention, but selection itself can still depend on context length. Recurrent layers retain a bounded summary instead of every token's KV. These mechanisms change different memory and compute terms. Combining them requires an engine that implements their cache layouts, prefix reuse and kernels correctly.

Latent attention changes the representation

A million-token request reaches a serving replica. Before asking how many positions attention visits, ask what the replica retains for each position. In conventional attention, stored key/value heads occupy memory. In Multi-Head Latent Attention, a shared latent vector encodes their content, with separate positional information where the architecture requires it.

A conceptual score can be rearranged:

content key for position j:  k_j = U_K c_j
content score:              qᵀ k_j = (U_Kᵀ q)ᵀ c_j
weighted values:            Σ a_j (U_V c_j) = U_V (Σ a_j c_j)

Here c_j is cached latent state, a_j an attention weight, and U_K/U_V learned projections. The rearrangement shows why a decode kernel need not reconstruct and store full K/V for every past token. Rotary terms, per-head projections and the output projection require the architecture's actual equations. The DeepSeek-V2 report describes MLA and projection absorption; this algebra is a simplified explanation, not a replacement implementation.

Latent attention is an architectural choice, not lossless compression you can apply to arbitrary GQA weights. Its efficiency depends on which operations the kernel fuses or absorbs. It does not universally add a fixed amount of arithmetic or guarantee a particular decode speedup.

Work the bytes with units and scope

From GLM-5.3's pinned config, a BF16 compressed-latent representation for its 78 main layers has:

per layer per position = (512 latent + 64 rotary) × 2 B = 1,152 B
across 78 layers       = 89,856 B = 87.75 KiB
at 1,048,576 positions = 94,220,845,056 B = 87.75 GiB

This excludes indexer and auxiliary state, padding, prediction layers and replication. A single logical cache copy is not automatically divided by tensor-parallel degree. The backend may replicate latent state across ranks while sharding other tensors. KV Cache Sizing covers that distinction.

For comparison, an explicitly assumed GQA shape with 80 layers, 8 KV heads, equal K/V width 128 and BF16 storage uses 2 × 80 × 8 × 128 × 2 = 327,680 B, or 320 KiB per position. Different trained architectures, widths and layer counts make this a capacity comparison, not a controlled quality comparison. Latent cache size depends on latent width, positional width, depth and dtype; it does not depend on depth alone.

Selection does not make the whole operation constant-cost

Consider an indexer that scores all L retained positions cheaply and selects k for more expensive attention. A simple operation-cost model is:

indexed decode attention cost ≈ aL + bk + selection overhead
full decode attention cost    ≈ bL

Even with fixed k, the first expression grows with L. Summed across prefill positions, an all-history index can retain a quadratic term with a smaller coefficient. Compressed, hierarchical and windowed indexes need different formulas. Do not transfer one family's complexity to another from the word “sparse.”

For a synthetic comparison, let a=1, b=100, and k=2,048, ignoring selection overhead. These are cost units, not measured GPU timings:

A 512-fold reduction in fully attended positions at the last row therefore produces only an 83.66-fold reduction in this simplified cost. The index is doing work. Real kernels add selection, irregular reads and communication.

An unselected token may be selected by the next query, so skipping its expensive attention today does not justify freeing its state. Some architectures also compress or evict state according to a trained retention scheme. In those cases memory changes because of that additional mechanism, not because top-k selection alone proves the data unnecessary.

Hybrid state adds another lifetime

Kimi K3's configuration has 69 recurrent KDA and 24 gated MLA layers. Qwen3.8-27B combines 48 linear and 16 full-attention layers. Neither model has zero memory in its linear layers.

The entire stack must resume from a consistent prefix boundary. Reusing attention blocks at token 8,192 alongside recurrent state from token 7,000 is not a valid cache hit. Recompute the missing state or choose a common usable boundary. More snapshots can reduce recomputation while consuming capacity that could serve another request.

What changes the deployment decision

Prefer the supported architecture that meets task quality and latency within the available memory. Reverse a projected memory advantage if per-rank replication, recurrent snapshots or unsupported kernels erase it in the measured workload. Test short-context concurrency as well as long-context requests: bounded recurrent state can dominate before growing KV becomes large.

Key takeaways

• GLM's specified main-layer BF16 latent component is 87.75 KiB per position, before other state and placement.
• Top-k bounds selected attention work; an all-history index still costs a term proportional to context per decode query.
• Current recurrent state is bounded per sequence, but retained snapshots grow with saved boundaries.
• Compression, selection and recurrence are separate mechanisms. Budget and validate each one.