Your deployment is limited by KV cache rather than by compute. What are the options?

Confirm that requests are waiting for usable state capacity. Separate allocated memory, live blocks and evictable prefixes, then compare admission, lower-precision state, prefix sharing and hardware changes with explicit assumptions.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.

Foundational

🧮 Napkin Math & Capacity

KV Cache SizingDerive cache payload from the stored attention state, then account for mixed sequence lengths, block allocation and per-device limits. Worked examples distinguish exact bytes from deployable capacity.

Foundational

🧮 Open Weights & Serving Engines

Reading config.json to Size a Model You Have Never RunRead a model configuration as an architecture description, then combine it with tensor metadata and engine allocation rules. Learn which fields determine attention state, why mixed precision needs tensor-level accounting, and how to turn a first estimate into a testable per-GPU capacity plan.

Foundational

🧮 Open Weights & Serving Engines

Multi-Head Latent Attention and Sparse IndexersLatent attention changes what is stored; sparse indexing changes which positions receive expensive attention; recurrent layers summarize history in state. Learn to budget each mechanism separately, including indexer work, cache replication and hybrid prefix snapshots.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour p99 time to first token is four times p50. Find out why.Next in this trackYou have one week to get a newly released trillion-parameter model serving production traffic. Plan it.Next in this trackSelf-host a 753B open-weights model or call a hosted API? Work the crossover.

FEDITOR'S NOTE

Scored on allocation evidence rather than an SM-activity shortcut, paged-cache semantics, rank-local sizing, bounded precision gains and conditional prefix-sharing or capacity remedies.

DISCUSSION · 0

No comments yet — be the first to share your approach.