hardNewNVIDIAFireworksPerplexity

Design prefill and decode disaggregation for a 70B serving fleet: pool sizes, KV transfer, routing, and when to turn it on.

Prefill is compute-bound and decode is bandwidth-bound, and one GPU running both does each worse. The pool ratio from the traffic's prompt and output lengths, the KV transfer cost per request and the link it needs, the router that pairs a prefill worker with a decode worker, and the load at which it beats a monolith.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Foundational

🧮 Open Weights & Serving Engines

Multi-Node Serving TopologiesOnce a model needs more GPUs than one NVLink domain holds, the deployment shape becomes a real design decision. Tensor parallelism stays inside the node because it communicates twice per layer per token. Across nodes the choices are data parallelism with replicas, pipeline parallelism with a bubble, expert parallelism with an all-to-all, or disaggregation that runs prefill and decode on separate pools and ships the KV cache between them. Each has a different failure mode and a different scaling story.

Foundational

🚀 Inference & Serving

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Memory-Bound vs Compute-Bound KernelsEvery kernel is limited by one of two walls: how fast bytes arrive from HBM, or how fast the tensor cores can multiply. Which wall applies is decided by arithmetic intensity against the ridge point, and the two regimes need opposite fixes. Decode, LayerNorm and softmax are memory-bound; prefill GEMMs are compute-bound; the interview question is which one you are looking at and what you would do about it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.Next in this trackDesign inference for a 1T-parameter mixture-of-experts model: expert parallelism on an NVL72 rack, the attention cache, balance.Next in this trackDesign a training cluster for a one-trillion-parameter MoE. Size it, choose the parallel layout, and map it onto the fabric.

FEDITOR'S NOTE

Scored on deriving the prefill-to-decode pool ratio from token counts and per-pool throughput, on the KV transfer arithmetic (bytes per request, link, overlap), on the two-hop routing with failure handling, and on stating the traffic and scale where disaggregation wins.

DISCUSSION · 0

No comments yet — be the first to share your approach.