Preparing for a platform or ML-infrastructure loop
Design rounds now ask how you would route, split and scale LLM serving on a cluster, not only on one node. Chapter 24 runs twelve rounds with the numbers to compute and what strong answers contain.
Handbooks 02 / September 2026 / Premium PDF
Distributed Inference on Kubernetes
Dynamo, llm-d and disaggregated serving: how to route, split, place, scale and operate LLM serving across a cluster.
Start from what a Deployment and a Service get wrong about LLM serving. Score pods on cache and load, split prefill from decode and size both pools, move the KV cache over RDMA, gang-schedule a multi-node replica, then deploy, secure, benchmark and migrate with NVIDIA Dynamo and llm-d. Twenty-five chapters, with the arithmetic worked at every step.
The complete PDF and new editions are included while your Premium membership is active. Keep the copies you download.
Already a member? Sign in to download your copy →
96illustrated pages, 25 chapters
23diagrams drawn from the worked numbers
12 + 69mock design rounds and self-check questions with answers
Read three pages before you decide
See how each chapter works through the numbers.
These are real pages from the PDF: how llm-d and Dynamo score pods on cache and load, why the prefill-to-decode ratio flips with the traffic mix, and how DeepSeek-V3 spreads across nodes. Each has a text version.
Read this page as text
Two ways to score a pod: llm-d's weighted sum and Dynamo's block cost
- With illustrative weights (prefix 3, queue 2, KV utilization 2), pod A holds the prompt prefix but is busy and scores 3.7; nearly idle pod B scores 3.9 and wins. Raise the prefix weight to 4 and pod A wins, 4.6 to 4.0.
- Dynamo's router counts everything in KV blocks: adjusted prefill is max(0, active prefill blocks + incoming prompt blocks − overlap credit), potential decode is active decode blocks + incoming blocks, and the lowest total cost wins.
- Worked with 16-token blocks: a 4,096-token prompt is 256 blocks. Worker A already holds 240 of them but is loaded, costing 416 + 2,056 = 2,472. Worker B holds none and costs 1,712, so the colder worker wins.
Read this page as text
The prefill-to-decode ratio flips with the traffic mix
- At 40 requests per second on the same model and hardware, a retrieval-heavy workload (4,000-token prompts, 250-token outputs) needs 6 prefill nodes and 2 decode nodes.
- A chat workload (1,000-token prompts, 600-token outputs) needs 2 prefill nodes and 5 decode nodes.
- If half the retrieval prompt tokens hit a prefix cache, prefill demand halves and 3 prefill nodes suffice, so a routing change that raises the hit rate is a capacity change.
Read this page as text
DeepSeek-V3 across two nodes, and what its authors deployed
- On sixteen H100s, each GPU holds about 41.9 GB of expert weights and about 26.1 GB of cache for its own sequences under data-parallel attention: roughly 371,000 tokens per GPU, 5.9 million per replica.
- The DeepSeek-V3 report describes a 4-node, 32-GPU prefill unit (EP32) and a 40-node, 320-GPU decode unit (EP320), exchanging tokens over InfiniBand.
- The phases want different layouts at different scales, and expert load is uneven, so the authors periodically duplicate high-load experts.
Who it is for
Written for the design round, and for the cluster after it.
A Kubernetes engineer new to LLM serving
You know Deployments, Services and the HPA. The first chapter shows the four assumptions they make that LLM serving breaks, and the rest of the handbook builds the replacements one at a time.
Choosing between Dynamo, llm-d or your own stack
Both projects are covered end to end with the same serving graph, then compared by the questions that actually decide it: engines, how much you want inside one controller, autoscaling and lock-in at the routing layer.
Operating a disaggregated fleet
Debugging by symptom, the runbook, rollouts that keep rollback at the routing boundary, and the RDMA checks that stop a fabric misconfiguration from quietly turning a 3 ms transfer into hundreds of milliseconds.
Contents
Twenty-five chapters, one serving graph.
The problem
- What a Deployment and a Service get wrong about LLM serving
- The serving graph, and how Dynamo and llm-d draw it
Routing
- The Gateway API Inference Extension: InferencePool and the Endpoint Picker
- Scoring pods on cache and load, with the arithmetic
- Queue at the gateway: flow control, priority and fairness
Disaggregation
- Two request flows for disaggregation, and when to use neither
- Moving KV cache between pods: NIXL, RDMA and the arithmetic
- Sizing prefill and decode pools from the traffic mix
Placement
- Replicas that span pods: LeaderWorkerSet, Grove and gang scheduling
- Placing GPU pods: fragmentation, topology and Dynamic Resource Allocation
- Serving a large mixture-of-experts model across nodes
Operating
- Autoscaling each role of the graph
- Model loading and cold starts
- Seeing one request across five hops, and debugging by symptom
- Rollouts, failure domains and recovery
The two stacks
- llm-d end to end: objects, paths and one request, hop by hop
- NVIDIA Dynamo end to end: the graph resource, the planner and one request, hop by hop
Deploying
- Deploying Dynamo: from an aggregated baseline to prefill and decode
- Deploying llm-d, and a bare prefill and decode reference
Securing and proving
- Trust zones, hardening and multi-tenancy
- Benchmarks you can defend, and what published results show
Deciding
- Dynamo, llm-d or plain replicas: making the call
- Migrating step by step, and the runbook to keep
Practice
- Twelve mock design rounds for platform and serving loops
Reference
- Formula sheet, resource map and glossary
A calculation from Chapter 8
How many prefill and decode nodes does a workload need?
A forward pass costs about 2 × 70 × 10⁹ FLOPs per token for Llama 3.1 70B. At an assumed 50 percent of eight H100s' 989 dense BF16 teraFLOPS, one TP8 node prefills 0.5 × 8 × 989 × 10¹² ÷ (2 × 70 × 10⁹) ≈ 28,257 prompt tokens per second.
At 40 requests per second with 4,000-token prompts, prefill demand is 160,000 tokens per second: 160,000 ÷ 28,257 = 5.66, so 6 nodes. Decode holds 40 × 250 × 0.025 = 250 sequences at a 25 ms inter-token target, and at about 130 sequences per node that is 2 nodes. The same model serving chat, with 1,000-token prompts and 600-token outputs, needs the ratio reversed: 2 prefill to 5 decode.
Size the cache in the KV-cache calculator → · Read the disaggregation concept →
Interview questions it prepares you for
Practice the prompts, then check the reasoning.
Each prompt below has its own worked answer on the site. The handbook gives the cluster-level reasoning that connects them.
- Design prefill and decode disaggregation for a 70B fleet →
- When does splitting prefill and decode onto separate pools pay? →
- Route requests across replicas for prefix hit rate →
- Design an internal LLM gateway →
- Design inference for a 1T-parameter mixture-of-experts model →
- Serve DeepSeek-V3: what changes versus a dense model? →
- Autoscale GPU replicas on queue depth without thrashing →
- The error rate spiked after a deploy. Find the cause. →
How it was made
Every number is derived, and every claim has a source.
The handbook cites 77 sources by number: NVIDIA Dynamo and llm-d documentation and release notes, the Gateway API Inference Extension specification, Kubernetes, LeaderWorkerSet, Kueue and KubeRay documentation, the DeepSeek-V3 report, and papers including DistServe, Splitwise and Mooncake. Its worked examples and self-check answers are recomputed by the build and must match the text before an edition ships.
Every chapter opens with links to the concept pages and questions that go deeper:
Request routing for LLMsDisaggregated prefill and decodePrefix cachingMulti-node serving topologiesTopology-aware communicationExpert parallelismAutoscaling and cold startsServing benchmarksInference platform architecture
Reviewed 2026-09-24. Calculations are first-order estimates with stated assumptions; no GPU benchmark was run. Manifests show the shape of a deployment; pin every version to a release you have tested. The cover is conceptual vector artwork.
Questions
Before you download
How do I get the PDF?
The complete PDF is included with active Premium membership. Sign in, then download it from this page. You keep the copies you download, and new editions are available while your access is active. The three sample pages above are free to read.
Which versions does it cover?
This edition was checked against NVIDIA Dynamo v1.5.0 and its nvidia.com/v1beta1 API, llm-d v0.9.0 and its release-0.9 guides, and the Gateway API Inference Extension v1 API. Both projects move quickly, so every chapter cites the maintainer page it relies on, and the deployment chapters tell you to pin versions to a release you have tested.
Do I need the first handbook?
It helps. The LLM Inference Systems Design Handbook explains what happens inside one engine: the KV cache, batching, scheduling and parallelism. This handbook assumes those ideas and covers running many engines on Kubernetes as one system. A glossary defines every term it uses.
Are the numbers benchmarks?
No. Worked numbers are first-order calculations from published specifications and model configurations, with every assumption stated. Published results, such as llm-d's routing and cache-sharing measurements and the DistServe and Splitwise papers, are attributed to their authors with their conditions. No GPU benchmark was run for this edition.
Are the interview rubrics from real companies?
No. The rubrics in the mock rounds describe what strong answers tend to contain. They are a study aid written for this handbook, not any company's grading criteria.
Keep the other guides beside it.
The LLM Inference Systems Design Handbook covers what happens inside one serving engine. The AI Infrastructure Field Guide builds the foundations from the first tensor to distributed systems. The AI Accelerator Field Guide compares GPUs and TPUs and is free with an account. The Open Model Field Guide covers model releases, architectures and deployment paths.