KV Cache Interview Questions: The Arithmetic That Decides Your Serving Answer

If you are interviewing for an inference or serving role, the KV cache is the single highest-value topic to have cold. It shows up in the design round as the real capacity limit, in the napkin math round as a derivation, in the cost round as the reason your dollars per million tokens are what they are, and in the debugging round as the reason your throughput collapsed when the average conversation got longer. OpenAI's reported design prompt for serving a chat model names KV cache memory math explicitly among the expectations.

The formula, and why it looks like that

Bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element

Every token that has been processed needs its key and value vectors kept, for every layer, because every future token attends back to it. That is the whole derivation, and being able to state it that way is better than reciting it.

The 2 is the key and the value. Layers because each has its own attention and its own cache. kv_heads, and this is where candidates go wrong. With grouped-query attention many query heads share a single key-value head. If a model has 64 attention heads and 8 key-value heads, the cache uses 8, not 64. Getting this wrong overestimates by that ratio. head_dim is the per-head dimension, often the model dimension divided by the attention head count. bytes_per_element is 2 for BF16 or FP16, 1 for FP8, 0.5 for INT4.

Worked: 80 layers, 8 kv heads, head dimension 128, BF16. That is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, which is 320 KiB per token exactly, or 327.68 KB in decimal. On a page about unit discipline the difference is worth saying out loud.

A 4,000-token conversation is about 1.3 GB. A 32,000-token context is about 10.5 GB, for one request. On a GPU with 80 GB where the weights take 40, you have 40 GB left, so roughly 30 short conversations or fewer than 4 long ones. That number is your concurrency limit, and the KV cache calculator will check the arithmetic while you practise.

Why this dominates the weights

Weights are a fixed cost. You pay 140 GB once for a 70B model in BF16, whether you serve one user or a thousand. The KV cache is per concurrent request and linear in sequence length.

So the capacity question is never "does the model fit." It is "what is left after the model, and how many requests does that buy at my actual context length." On long-context workloads the aggregate cache routinely exceeds the weights, and a candidate who has only ever sized weights will be off by a large factor.

Take it to cost. If the cache caps you at 30 concurrent requests instead of 60, your throughput per GPU roughly halves and your cost per million tokens roughly doubles. The cost per token calculator closes that loop, and the napkin math track drills the chain.

The six levers, in the order they are worth discussing

1. Grouped-query or multi-query attention. The largest architectural lever, cutting the cache by the ratio of query heads to key-value heads. It is a model design decision, so in a serving round you discuss it as a constraint you inherited or a requirement you would push back to the model team.

2. Quantise the cache. BF16 to FP8 halves it, and typically costs little quality. INT4 halves it again with more risk, particularly on long contexts. Name the trade and say how you would validate it, rather than asserting it is free.

3. Paged attention. Contiguous allocation sized for the worst case wastes most of the allocation, because most requests are much shorter than the maximum. Storing the cache in fixed-size blocks with an indirection table, exactly as virtual memory works, allocates per block as the sequence grows and largely removes that internal fragmentation. This is one of the biggest practical wins in modern serving.

4. Prefix sharing. In a chat product every request carries the same system prompt, sometimes thousands of tokens of it. With paged storage those blocks can be shared across concurrent requests rather than duplicated. This is close to free capacity and very few candidates mention it unprompted.

5. Eviction, offload and recompute. Move cold blocks to host memory, or drop them and recompute on demand. Say the cost honestly: recomputing a dropped prefix is a prefill, and prefill is compute-bound, so this trades a memory limit for a compute limit. Choosing which resource you would rather be limited by is the actual answer.

6. Admission control. When the cache is full, the right behaviour is to queue or reject rather than to thrash. A system that admits everything and then evicts under pressure produces worse tail latency than one that says no at the door.

Our inference and serving questions work each of these to the depth a round demands.

Where it shows up in each round

Design. "Design infrastructure for serving a chat model." The KV chain gives you concurrency per GPU, which gives you fleet size, which gives you cost. Run it out loud, as in the GPU cluster design round.

Napkin math. A direct derivation, sometimes with a comparison between multi-head and grouped-query attention.

Debugging. "Throughput dropped 40 percent this week and nothing was deployed." A plausible answer: the average context length grew, so the cache per request grew, so fewer requests fit, so the effective batch size fell, so decode lost its batching efficiency. That chain is worth being able to say fluently.

Trade-off questions. "How would you double the number of users on the same hardware?" Quantise the cache, share prefixes, check for fragmentation, then consider offload with the recompute cost named.

The three mistakes to avoid

Using the attention head count instead of the key-value head count. Sizing only the weights and forgetting the cache entirely. And treating eviction as free when it is a prefill in disguise.

Get the formula automatic, then practise saying the chain from bytes to dollars without stopping. Start with the inference and serving track, keep the formula sheet open, and use the must-know questions to find what is still slow.