Handbooks 02 / Version 2 · October 2026 / Premium PDF

Distributed Inference on Kubernetes

Dynamo, llm-d and disaggregated serving: how to route, split, place, scale and operate LLM serving across a cluster.

Start from what a Deployment and a Service get wrong about LLM serving. Score pods on cache and load, split prefill from decode and size both pools, move the KV cache over RDMA, gang-schedule a multi-node replica, then deploy, secure, benchmark and migrate with NVIDIA Dynamo and llm-d. Twenty-five chapters, with worked calculations, deployment examples and operational checks.

Get paid Premium →Read three sample pages ↓

New to LLM infrastructure? Start with the free learning path →

Premium guide PDFs and companion files require an active paid Premium membership or approved complimentary access. Referral Premium includes the rest of the site, but excludes these downloads. Keep the copies you download.

Already a member? Sign in to download your copy →

Distributed Inference on Kubernetes cover: a grid of cluster nodes whose pods are colored by serving role, joined by request paths

100illustrated pages, 25 chapters

23diagrams drawn from the worked numbers

12 + 69mock design rounds and self-check questions with answers

Read three pages before you decide

See how each chapter works through the numbers.

These are real pages from the PDF: how llm-d and Dynamo score pods on cache and load, why the prefill-to-decode ratio flips with the traffic mix, and why replicated model weights can overturn a GPU memory estimate. Each selected page has its complete text and diagram description below.

Sample page 14

Scoring pods on cache and load, with the arithmetic

View the printed page ↗

What the diagram shows: llm-d style weighted scoring with illustrative weights (prefix 3, queue 2, KV utilization 2). Pod A holds the prefix but is busy; pod B is nearly idle and wins, 3.9 to 3.7. Inputs (prefix, queue, KV) are A = (0.9, 0.2, 0.3) and B = (0.1, 0.9, 0.9). With prefix × 3, queue × 2 and KV utilization × 2, Pod A gets prefix 2.7 + queue 0.4 + KV 0.6 = 3.7. Pod B gets prefix 0.3 + queue 1.8 + KV 1.8 = 3.9 and wins. Raise the prefix weight to 4 and pod A wins, 4.6 to 4.0: the weights encode how much queueing a cache hit is worth.

The example shows the point of weighting. Cache affinity is worth a lot, and it still loses to a pod that can start the request now. Change the prefix weight to 4 and pod A wins, 4.6 to 4.0. Weights are therefore a product decision about how much queueing a cache hit is worth, and they should be tuned against measured TTFT, not guessed.

llm-d also offers a prefix-cache-affinity-filter that narrows candidates to "sticky" pods with high prefix scores, with a TTFT load gate that breaks stickiness when those pods become much slower than the rest.[13] That is the hot-key defense built into the filter stage.

Dynamo: one cost in blocks

Dynamo's router expresses everything in KV blocks and picks the lowest cost.[4] In its documented form:

Dynamo router cost.

adjusted prefill = max(0, active prefill blocks + incoming prompt blocks − overlap credit blocks). potential decode = active decode blocks + incoming blocks. cost = prefill load scale × adjusted prefill + potential decode + an optional per-request term.

The overlap credit is the part of the prompt a worker already holds, in device memory or in configured host, disk or shared tiers. The per-request term is off by default.

Worked example, 16-token blocks, prefill load scale 1.

A 4,096-token prompt is 256 blocks. Worker A holds 240 of them, but already has 400 prefill blocks and 1,800 decode blocks in flight: max(0, 400 + 256 − 240) + (1,800 + 256) = 416 + 2,056 = 2,472. Worker B holds none and has 1,200 decode blocks in flight: 256 + (1,200 + 256) = 1,712. Worker B wins.

Sample page 29

Sizing prefill and decode pools from the traffic mix

View the printed page ↗

	Retrieval-heavy	Chat
Prompt / output tokens	4,000 / 250	1,000 / 600
Prefill tokens per second	160,000	40,000
Prefill nodes	160,000 ÷ 28,257 = 5.66, so 6	40,000 ÷ 28,257 = 1.42, so 2
Sequences decoding	40 × 250 × 0.025 = 250	40 × 600 × 0.025 = 600
Decode nodes	250 ÷ 130 = 1.92, so 2	600 ÷ 130 = 4.6, so 5
Ratio	6 prefill : 2 decode	2 prefill : 5 decode

What the diagram shows: Prefill and decode nodes for two workloads at 40 requests per second on the same model and hardware. The ratio flips with the prompt-to-output mix. Retrieval-heavy: 4,000-token prompts, 250-token outputs, 6 prefill and 2 decode nodes (6 : 2). Chat: 1,000-token prompts, 600-token outputs, 2 prefill and 5 decode nodes (2 : 5). Both at 40 requests per second and a 25 ms ITL target.

The same model on the same hardware needs a prefill-heavy fleet for one product and a decode-heavy fleet for another. A fixed ratio chosen at launch will be wrong the day the traffic mix shifts, which is why both projects scale the pools separately and why Dynamo supports adding and removing prefill and decode workers at runtime.[2]

Prefix reuse moves the ratio

Cached prompt tokens do not need prefill. If half of the retrieval workload's prompt tokens hit a cache, prefill demand halves: 80,000 ÷ 28,257 = 2.83, so 3 nodes instead of 6. Dynamo's planner takes the observed KV hit rate as one of its traffic features for exactly this reason.[6] A routing change that raises the hit rate is a capacity change, and should be planned as one.

Sample page 39

Serving a large mixture-of-experts model across nodes

View the printed page ↗

The per-rank budget.

KV room = usable GPU bytes − expert bytes assigned to this rank − non-expert bytes assigned to this rank − runtime reserve. For an ideal even EP split with TP1 non-expert replication, this becomes usable − expert bytes / EP − replicated bytes − reserve. Account separately for redundant experts and unequal shard sizes; the most constrained rank limits the layout.

Consider a deliberately hypothetical 671 GB inventory: 640 GB of sharded expert weights and 31 GB of fully replicated non-expert weights. At EP16/TP1, each GPU needs 640 / 16 + 31 = 71 GB for weights. With 72 GB usable and a 4 GB runtime reserve, KV room is −3 GB: the layout fails before caching one token. Dividing the whole 671 GB by sixteen would instead suggest 26.0625 GB of KV room. This invented partition teaches the accounting error; it is not a measurement of DeepSeek's tensor inventory.

Cache duplication is a separate comparison. Hold usable KV room at an assumed 20 GB per rank after the weight calculation. At 70,272 bytes per token, each rank holds floor(20,000,000,000 / 70,272) = 284,608 token positions before block rounding and other cache overhead. If an MLA implementation duplicates each sequence's full latent on sixteen TP ranks, the group still holds 284,608 unique positions. Sixteen independent TP1 attention ranks could hold 16 × 284,608 = 4,553,728 positions across balanced sequence shards. That factor of sixteen assumes equal KV room and the stated cache layout; changing attention parallelism also changes weight memory, so it is not a measured capacity ratio for sixteen H100s.

What the diagram shows: A hypothetical EP16, TP1 layout: 640 GB of expert weights are sharded, while 31 GB of other weights are replicated on every rank. Each GPU needs 75 GB before KV cache, exceeding a 72 GB usable budget. Dividing all 671 GB by sixteen would incorrectly suggest a fit. This is a counterexample, not a measured DeepSeek deployment. Count replicated weights on every GPU. The hypothetical EP16 / TP1 layout uses decimal GB: expert shard 640 ÷ 16 = 40 GB, replicated weights 31 GB, and a 4 GB runtime reserve shown by the gray segment. The dashed line marks the 72 GB usable limit. The per-GPU calculation is 72 − 40 − 31 − 4 = −3 GB left for cache. This layout does not fit, even before storing a single token of KV cache. The tempting shortcut, 72 − (671 ÷ 16) − 4, gives +26.1 GB. It is wrong here because the 31 GB is replicated, not divided by sixteen.

Who it is for

Written for the design round, and for the cluster after it.

Preparing for a platform or ML-infrastructure loop

Practice explaining how to route, split and scale LLM serving across a cluster. Chapter 24 runs twelve rounds with the numbers to compute and what strong answers contain.

A Kubernetes engineer new to LLM serving

You know Deployments, Services and the HPA. The first chapter shows the four assumptions they make that LLM serving breaks, and the rest of the handbook builds the replacements one at a time.

Choosing between Dynamo, llm-d or your own stack

Both projects are covered end to end with the same serving graph, then compared by the questions that actually decide it: engines, how much you want inside one controller, autoscaling and lock-in at the routing layer.

Operating a disaggregated fleet

Debugging by symptom, the runbook, rollouts that keep rollback at the routing boundary, and the RDMA checks that stop a fabric misconfiguration from quietly turning a 3 ms transfer into hundreds of milliseconds.

Contents

Twenty-five chapters, one serving graph.

The problem

What a Deployment and a Service get wrong about LLM serving
The serving graph, and how Dynamo and llm-d draw it

Routing

The Gateway API Inference Extension: InferencePool and the Endpoint Picker
Scoring pods on cache and load, with the arithmetic
Queue at the gateway: flow control, priority and fairness

Disaggregation

Two request flows for disaggregation, and when to use neither
Moving KV cache between pods: NIXL, RDMA and the arithmetic
Sizing prefill and decode pools from the traffic mix

Placement

Replicas that span pods: LeaderWorkerSet, Grove and gang scheduling
Placing GPU pods: fragmentation, topology and Dynamic Resource Allocation
Serving a large mixture-of-experts model across nodes

Operating

Autoscaling each role of the graph
Model loading and cold starts
Seeing one request across five hops, and debugging by symptom
Rollouts, failure domains and recovery

The two stacks

llm-d end to end: objects, paths and one request, hop by hop
NVIDIA Dynamo end to end: the graph resource, the planner and one request, hop by hop

Deploying

Deploying Dynamo: from an aggregated baseline to prefill and decode
Deploying llm-d, and a bare prefill and decode reference

Securing and proving

Trust zones, hardening and multi-tenancy
Benchmarks you can defend, and what published results show

Deciding

Dynamo, llm-d or plain replicas: making the call
Migrating step by step, and the runbook to keep

Practice

Twelve mock design rounds for platform and serving loops

Reference

Formula sheet, resource map and glossary

A calculation from Chapter 8

How many prefill and decode nodes does a workload need?

A forward pass costs about 2 × 70 × 10⁹ FLOPs per token for Llama 3.1 70B. At an assumed 50 percent of eight H100s' 989 dense BF16 teraFLOPS, one TP8 node prefills 0.5 × 8 × 989 × 10¹² ÷ (2 × 70 × 10⁹) ≈ 28,257 prompt tokens per second.

At 40 requests per second with 4,000-token prompts, prefill demand is 160,000 tokens per second: 160,000 ÷ 28,257 = 5.66, so 6 nodes. Decode holds 40 × 250 × 0.025 = 250 sequences at a 25 ms inter-token target, and an assumed operating point of 130 sequences per node gives 2 nodes. This is a planning example; benchmark that concurrency at your latency target. The same model serving chat, with 1,000-token prompts and 600-token outputs, needs the ratio reversed: 2 prefill to 5 decode.

Size the cache in the KV-cache calculator → · Read the disaggregation concept →

Interview questions it prepares you for

Practice the prompts, then check the reasoning.

Each prompt below has its own worked answer on the site. The handbook gives the cluster-level reasoning that connects them.

How it was made

Sources, calculations and review.

The handbook cites 82 sources by number: NVIDIA Dynamo and llm-d documentation and release notes, the Gateway API Inference Extension specification, Kubernetes, LeaderWorkerSet, Kueue and KubeRay documentation, the DeepSeek-V3 report, and papers including DistServe, Splitwise and Mooncake. The build checks selected numerical examples and text anchors. Separate reasoning checks cover memory replication, latency percentiles and shared download bandwidth. The conceptual answers also need editorial and source review; an arithmetic check cannot establish that a deployment works.

Every chapter opens with links to the concept pages and questions that go deeper:

Request routing for LLMs Disaggregated prefill and decode Prefix caching Multi-node serving topologies Topology-aware communication Expert parallelism Autoscaling and cold starts Serving benchmarks Inference platform architecture

What changed in Version 2

The 18 October update makes the prefill and decode pool-sizing assumptions explicit and corrects the diagram that labeled every prefill pool compute-bound. Chapter 8 now explains which workload and latency measurements must replace those planning estimates. Deployment recipes retain the documented September baseline: Dynamo v1.5.0 and llm-d v0.9.0. This update does not claim a new cluster qualification.

Reviewed 2026-10-18. Calculations are first-order estimates with stated assumptions; no GPU benchmark was run. Deployment examples received CPU syntax and manifest-schema checks. They were not executed on a GPU cluster. Verify your installed versions, fabric, readiness and complete responses before using them. The cover is conceptual vector artwork.

Questions

Before you download

How do I get the PDF?

The complete PDF requires active paid Premium membership or approved complimentary access. Referral Premium does not include this download. Sign in with an eligible account, then download it from this page. You keep the copies you download, and new editions are available while your access is active. The three sample pages above are free to read.

Which versions does it cover?

This edition was checked against NVIDIA Dynamo v1.5.0 and its nvidia.com/v1beta1 API, llm-d v0.9.0 and its release-0.9 guides, and the Gateway API Inference Extension v1 API. Both projects move quickly, so every chapter cites the maintainer page it relies on, and the deployment chapters tell you to pin versions to a release you have tested.

Do I need the first handbook?

It helps. The LLM Inference Systems Design Handbook explains what happens inside one engine: the KV cache, batching, scheduling and parallelism. This handbook assumes those ideas and covers running many engines on Kubernetes as one system. The glossary covers key terms including BF16, tensor parallelism, RDMA, TTFT and ITL. Start with the free learning path if these are new to you.

Are the numbers benchmarks?

No. Worked numbers are first-order calculations from published specifications and model configurations, with every assumption stated. Published results, such as llm-d's routing and cache-sharing measurements and the DistServe and Splitwise papers, are attributed to their authors with their conditions. No GPU benchmark was run for this edition.

Are the interview rubrics from real companies?

No. The rubrics in the mock rounds describe what strong answers tend to contain. They are a study aid written for this handbook, not any company's grading criteria.

Get paid Premium →Read the first handbook

Keep the other guides beside it.

The LLM Inference Systems Design Handbook covers what happens inside one serving engine. The AI Infrastructure Field Guide builds the foundations from the first tensor to distributed systems. The AI Accelerator Field Guide compares GPUs and TPUs and is free with an account. The Open Model Field Guide covers model releases, architectures and deployment paths.