hardAnthropicGoogleFireworks

You need to serve 128k-token contexts. What breaks first, and what do you change?

Calculate retained cache and prefill work for a stated 128k model shape, then choose scheduling and memory policies from measured limits.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Calculate retained cache and prefill work for a stated 128k model shape, then choose scheduling and memory policies from measured limits.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🧮 Open Weights & Serving Engines

Multi-Head Latent Attention and Sparse IndexersLatent attention changes what is stored; sparse indexing changes which positions receive expensive attention; recurrent layers summarize history in state. Learn to budget each mechanism separately, including indexer work, cache replication and hybrid prefix snapshots.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationCount the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

Advanced

🚀 Inference & Serving🔒 Premium

Chunked PrefillFollow a long prompt through a shared token budget, including its final partial chunk. Learn why chunking can reduce stream pauses without guaranteeing a fixed token gap or a universal throughput gain.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackEAGLE, Medusa or a separate draft model: which speculative decoding method do you pick, and why?Next in this trackDraw the throughput versus latency curve for LLM serving and explain where you would operate on it.Next in this trackDesign the fastest serving path for a 70B model with a 200 ms TTFT target at production scale.

FEDITOR'S NOTE

Derive cache demand from the selected model and reserve room for output. Diagnose memory, prefill and decode limits separately; a FLOP estimate does not predict a latency percentile.

DISCUSSION · 0

No comments yet — be the first to share your approach.