Your deployment is limited by KV cache rather than by compute. What are the options?

A full cache pool with idle streaming multiprocessors is a specific signature with a specific set of remedies. Six of them, ranked by what they cost, and the one that is free because the flag was set from the model rather than the product.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Foundational

🧮 Napkin Math & Capacity

KV Cache SizingThe KV cache is the memory that decides how many users a serving replica can hold and how long their context can be. Its size per token comes from four numbers in the model's config file (layers, KV heads, head dimension, bytes per element) and one formula; multiplied by context and concurrency it is the number every capacity plan is built on. This page derives it, works it for four models including an MLA one, and shows the two places candidates get it wrong by a factor of eight.

Foundational

🧮 Open Weights & Serving Engines

Reading config.json to Size a Model You Have Never RunEvery Hugging Face model ships a config.json, and it contains enough to compute the weight footprint, the KV cache per token, the parallel degrees that divide cleanly and the minimum GPU count, before downloading a byte. Doing that derivation is a standard whiteboard exercise in serving interviews because it is exactly what an engineer does on the morning a new model lands, and the fields that matter are the same across every recent architecture.

Foundational

🧮 Open Weights & Serving Engines

Multi-Head Latent Attention and Sparse IndexersGrouped-query attention shrank the KV cache by sharing key and value heads. Latent attention goes further by caching a single compressed vector per token per layer and reconstructing the heads on the fly, which cuts the cache by tens of times rather than by a small factor. On top of that, sparse indexers pick a few thousand relevant positions per query instead of attending to all of them, turning the quadratic term linear at long context. Both are now standard in open-weights models, and both change how a serving deployment is sized.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour p99 time to first token is four times p50. Find out why.Next in this trackYou have one week to get a newly released trillion-parameter model serving production traffic. Plan it.Next in this trackSelf-host a 753B open-weights model or call a hosted API? Work the crossover.

FEDITOR'S NOTE

Scored on recognizing the signature, on ranking remedies by cost with the free one first, and on the model-level option that changes KV per token by tens of times.

DISCUSSION · 0

No comments yet — be the first to share your approach.