← LLM Inference & Serving / 11

hard★ EssentialAnthropicFireworksNVIDIA

When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?

Compare a complete phase handoff with a tuned combined pool, then test the queueing reversal and qualified completions. Nominal link speed and fleet size alone cannot decide the split.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Compare a complete phase handoff with a tuned combined pool, then test the queueing reversal and qualified completions. Nominal link speed and fleet size alone cannot decide the split.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodeSeparate prefill and decode to control phase interference, then budget the KV handoff. A short isolated transfer can still overload a shared path; cancellation stops generation before it necessarily releases transfer buffers.

🧮 Open Weights & Serving Engines

Multi-Node Serving TopologiesChoose what to distribute: complete replicas, tensor operations, layers, experts or request phases. Compare rank-local fit, communication dependencies, workload concurrency and failure domains. A node boundary is not a universal performance boundary, and multiple layouts can be combined.

🚀 Inference & Serving

Prefill vs DecodeFollow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputRead a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackEvery request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?Next in this trackYou have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.Next in this trackYour p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.

FEDITOR'S NOTE

Work actual state, timing boundaries and a matched outcome comparison; explain which measurement would reverse the decision.

DISCUSSION · 0

No comments yet — be the first to share your approach.