The model advertises a million tokens of context. What actually breaks when you serve it?

A context-window limit is not a latency or concurrency guarantee. Budget retained state, prefill and shared service time; distinguish cache reuse from eliminating work; choose isolation only when measured interference justifies its cost.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🧮 Open Weights & Serving Engines

Multi-Head Latent Attention and Sparse IndexersLatent attention changes what is stored; sparse indexing changes which positions receive expensive attention; recurrent layers summarize history in state. Learn to budget each mechanism separately, including indexer work, cache replication and hybrid prefix snapshots.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationCount the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodeSeparate prefill and decode to control phase interference, then budget the KV handoff. A short isolated transfer can still overload a shared path; cancellation stops generation before it necessarily releases transfer buffers.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackThe model authors publish a new revision. How do you roll it out without a quality regression?Next in this trackDesign a platform that serves twenty open-weights models of varying size to internal teams.Next in this trackAn SGLang deployment underperforms. Tune it against the project's own published targets.

FEDITOR'S NOTE

Token count arrives after parsing and tokenization. A huge upload can exhaust gateway memory before it becomes a GPU request, so give the ingress path its own byte and processing limits.

DISCUSSION · 0

No comments yet — be the first to share your approach.