KV Cache Interview Questions: Derive Memory, Then Test Capacity
Derive Llama 3.1 70B KV-cache memory with explicit units, check a per-GPU budget, and explain paging, prefix reuse and quantization without confusing memory capacity with throughput.
BY MAYA CASTILLO · AIINFRAINTERVIEWS EDITORIAL · UPDATED SEPTEMBER 26, 2026 · 9 MIN READ
PRACTICE THIS:Inference and serving questions ·KV cache calculator ·Napkin math questions ·The formula sheet
A useful KV-cache interview answer derives the retained-state bytes, checks the budget on each GPU, and explains which measurement would establish serving capacity. For Llama 3.1 70B with BF16 cache, the raw full-attention state is 327,680 bytes per token. At 8,192 retained tokens, one sequence needs about 2.68 GB across the replica, before allocation overhead. That is a memory calculation. It does not establish tokens per second or a latency target.
Use the KV-cache calculator to vary the inputs while working through the questions below. These are preparation exercises, not claims that a particular company asked them.
Derive the bytes from the model
Keys and values let an attention layer reuse state from earlier tokens during generation. For full-attention MHA or GQA, each retained token has a key vector and a value vector for every KV head in every layer:
bytes per token = 2 × layers × KV heads × head dimension × bytes per element
= 2 × 80 × 8 × 128 × 2
= 327,680 bytes
= 320 KiB
= 327.68 decimal KB
The published Llama 3.1 70B configuration gives 80 layers, 64 query heads, eight KV heads and a hidden size of 8,192. For this architecture, dividing 8,192 by 64 gives a head dimension of 128. When another model explicitly supplies a head dimension, read it rather than assuming that quotient applies.
Using 64 query heads in place of eight KV heads would overestimate this cache by a factor of eight. The weight format and cache format are separate choices: an FP8 weight checkpoint can still use a two-byte cache. State the cache dtype before substituting its element size.
Turn the result into a memory budget
Consider a hypothetical replica with two devices, each offering 80 decimal GB. Use the calculator's rounded 70.6-billion-parameter preset, two-byte weights and an illustrative 8 GB runtime reserve per device. Assume an even two-way tensor split for this exercise; verify the actual engine layout before applying it.
raw weights across replica = 70.6 × 10^9 × 2 = 141.2 GB
remaining aggregate budget = 160 − 141.2 − 16 = 2.8 GB
KV for one 8,192-token sequence = 327,680 × 8,192
= 2,684,354,560 bytes
≈ 2.684 GB
memory-only sequence limit = floor(2.8 / 2.684) = 1
Under the assumed even split, each device holds 70.6 GB of weights and about 1.342 GB of that sequence's KV, leaving roughly 0.058 GB after its 8 GB reserve. The small margin makes this a useful sanity check: a modest uncounted allocation can overturn the estimate. Nominal device capacity and rounded parameter counts are not precise runtime measurements.
At 32,000 retained tokens the same raw cache is 10.486 GB per sequence across the replica, so it no longer fits this budget. If the two ranks allocate different amounts, the fuller rank sets the limit; adding their free memory together cannot rescue it. The worked Llama memory answer develops the placement check further.
Explain what the cache formula leaves out
The formula above assumes every listed layer retains full K and V tensors for every token counted. Sliding-window layers can retain a shorter window. Hybrid models can combine attention with recurrent state. Multi-head latent attention stores a compressed representation with different dimensions. Read the architecture before applying a familiar formula.
Logical token state is also different from physical allocation. A serving engine may allocate a KV pool at startup and assign blocks from it as requests arrive. Changes in occupied blocks need not appear as matching changes in total GPU memory reported by a device monitor. Separate weights, the KV pool and other runtime buffers when reconciling observations with the estimate.
Choose a way to reduce pressure
| Option | What changes | What can reverse the decision |
|---|---|---|
| Different model architecture | KV geometry, weights and model behavior | The alternative fails the quality or compatibility requirement |
| Lower-precision cache | Raw bytes per element | Quality loss, unsupported kernels or poor measured performance |
| Paged allocation | How growing sequences obtain physical blocks | Allocation was not the limiting waste in this workload |
| Prefix reuse | Repeated prefix state and prefill work | Few reusable tokens, eviction or incompatible cache identities |
| Offload or recompute | Where state is kept or how it is recovered | Transfer/recovery time breaks the latency target |
| Admission limits | Which requests enter the active batch | Queueing or rejection violates the service requirement |
The PagedAttention paper describes block-based KV management and sharing. Paging avoids requiring one large contiguous allocation per request, but the last partly used block and other allocation costs remain. Sharing requires compatible state; paging alone does not guarantee a prefix-cache hit.
For quantization, compare equivalent tensor layouts. FP8 halves the raw element bytes of BF16; metadata, padding and layers kept in another format change the total saving. vLLM's quantized-cache documentation describes calibration and backend constraints. Test the output quality at the contexts your application actually uses.
vLLM prefix caching reuses compatible prefix state to avoid repeating prefill work. It does not remove the work of generating subsequent tokens. Measure reusable prefix length and residency rather than assuming every chat request shares its entire system prompt. Keeping old prefixes resident also consumes cache capacity.
Offloading spends transfer time; recomputing spends recovery work. Neither implies a universal compute bottleneck. Compare the actual recovery path against the service's queueing and response-time limits. If recovery repeatedly causes latency failures, admitting fewer simultaneous requests can be the better policy.
Connect memory to throughput and cost carefully
Suppose a change raises the memory-only concurrency cap from 30 to 60. That establishes room for more retained state. It does not establish twice the throughput: the service may already be compute-limited, the extra sequences may lengthen each decode step, or there may not be enough demand to fill the batch.
Measure successful output under the same workload and latency requirements, including failed requests. As a hypothetical cost example, a $20/hour fleet producing 2,000 accepted output tokens per second costs:
cost per million accepted tokens = 20 × 1,000,000 / (3,600 × 2,000)
≈ $2.78
If measured accepted throughput falls to 1,000 tokens per second at the same fleet cost, this metric doubles to about $5.56. The measured rate establishes that relationship; a memory cap does not. The cost-per-token calculator and inference economics workbench let you explore the arithmetic with explicit assumptions.
Practise a follow-up that changes the answer
Start with the two-device example, then change the context or cache precision. Explain the new memory result before proposing a serving configuration. Ask what the engine logs say about available blocks, whether a rank is tighter than the others, and how the request-length distribution changes occupied state.
For a hypothetical throughput regression, growing contexts are one possible cause. Check retained tokens, active sequences, preemption and latency before concluding that KV capacity is responsible. Keep the formula sheet beside the inference and serving questions, and practise separating a calculated limit from a demonstrated operating point.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
For full-attention MHA or GQA, raw bytes per retained token are 2 × layers × KV heads × head dimension × bytes per element. Multiply by the retained tokens across sequences in one replica. Sliding-window, hybrid and MLA architectures require different state accounting; distributed placement determines each GPU's share.
It limits admission when the required cache exceeds the memory left after weights and runtime allocations. Weights or other buffers can also prevent a model from fitting. A memory limit alone does not predict throughput, latency or cost per token.
Query heads share key and value heads. With the same layers, head dimension and dtype, eight KV heads need one eighth of the raw cache of 64 KV heads. Use the actual model configuration; GQA is an architectural choice, not a serving flag that preserves arbitrary model weights.
It lets a sequence use noncontiguous fixed-size blocks and acquire more as it grows. A partially filled final block still wastes space, and runtime metadata and retained cached prefixes occupy memory. Logical sequence use and the engine's allocated KV pool are different quantities.
Consider it when cache capacity is a measured constraint and the model, backend and hardware support the format. FP8 uses half the raw bytes of BF16 for the same tensor shape, before scales and other allocations. Evaluate quality and serving performance before deciding that the additional capacity is useful.
Discussion (5)
Prompt tokens are only the starting balance. If admission reserves room for the requested output, the budget has to include that future growth too. Otherwise several requests can fit at admission and collide later.
And a scheduler that permits preemption is making a different promise. Compare those policies at the same completion and latency target, or the one admitting more requests can look better while finishing less useful work.
I would keep the billing denominator beside the memory result. Retried output can increase generated tokens without increasing accepted output. A lower dollar-per-generated-token number can hide a worse cost per successful task.
A shared prompt can stop being shared after one changed token near the beginning. Put stable material first when the application allows it, then measure reusable prefix length. A nominally long system prompt says little about actual hits.
Check where offloaded blocks land. A transfer test with idle host memory does not describe a server where checkpoint traffic and other replicas share the same path. The contention is part of the experiment.