← AI Infrastructure System Design / 28

hardNVIDIAFireworksPerplexity

Design prefill and decode disaggregation for a 70B serving fleet: pool sizes, KV transfer, routing, and when to turn it on.

Size both serving pools from workload-matched rates, then check memory, shared transfer paths and surviving capacity. Follow admission and buffer ownership through failure and scaling.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Size both serving pools from workload-matched rates, then check memory, shared transfer paths and surviving capacity. Follow admission and buffer ownership through failure and scaling.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodeSeparate prefill and decode to control phase interference, then budget the KV handoff. A short isolated transfer can still overload a shared path; cancellation stops generation before it necessarily releases transfer buffers.

🧮 Open Weights & Serving Engines

Multi-Node Serving TopologiesChoose what to distribute: complete replicas, tensor operations, layers, experts or request phases. Compare rank-local fit, communication dependencies, workload concurrency and failure domains. A node boundary is not a universal performance boundary, and multiple layouts can be combined.

🚀 Inference & Serving

Prefill vs DecodeFollow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.

🧩 GPU & Accelerator Architecture🔒 Premium

Memory-Bound vs Compute-Bound KernelsUse a worked timing example to distinguish a bandwidth ceiling from a measured bottleneck. Learn which profiler evidence to inspect and why decode, prefill and a whole serving step need separate diagnoses.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.Next in this trackDesign inference for a 1T-parameter mixture-of-experts model: expert parallelism on an NVL72 rack, the attention cache, balance.Next in this trackDesign a training cluster for a one-trillion-parameter MoE. Size it, choose the parallel layout, and map it onto the fabric.

FEDITOR'S NOTE

Derive coupled phase capacity and failure headroom, account for transfer and per-rank state, and justify a reversible rollout.

DISCUSSION · 0

No comments yet — be the first to share your approach.