Design a deployment that serves a trillion-parameter model at a million tokens of context with usable latency.

Every constraint in this bank meets in one design and they conflict. What has to be true of the model before the deployment is possible at all, the four mechanisms that make the latency usable, and the honest statement of what the first request still costs.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧮 Open Weights & Serving Engines

Multi-Head Latent Attention and Sparse IndexersGrouped-query attention shrank the KV cache by sharing key and value heads. Latent attention goes further by caching a single compressed vector per token per layer and reconstructing the heads on the fly, which cuts the cache by tens of times rather than by a small factor. On top of that, sparse indexers pick a few thousand relevant positions per query instead of attending to all of them, turning the quadratic term linear at long context. Both are now standard in open-weights models, and both change how a serving deployment is sized.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverRunning inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.

Foundational

🧮 Open Weights & Serving Engines

Multi-Node Serving TopologiesOnce a model needs more GPUs than one NVLink domain holds, the deployment shape becomes a real design decision. Tensor parallelism stays inside the node because it communicates twice per layer per token. Across nodes the choices are data parallelism with replicas, pipeline parallelism with a bubble, expert parallelism with an all-to-all, or disaggregation that runs prefill and decode on separate pools and ships the KV cache between them. Each has a different failure mode and a different scaling story.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackThe open-weights model you are deploying is multimodal. What changes about serving it?Next in this trackYour agent product's tool-call success rate fell from 94 to 61 percent after a deploy. Debug it.Next in this trackDesign the evaluation you run against a serving deployment, not against a model.

FEDITOR'S NOTE

Scored on the model's architecture being a precondition, on the four mechanisms with their arithmetic, and on being honest that first-request latency cannot be made interactive.

DISCUSSION · 0

No comments yet — be the first to share your approach.