Preparing for an inference or ML-infrastructure loop
The design round asks you to size a fleet, choose a batching policy and defend a latency target. Most chapters end with how to say it in the room, and Chapter 16 runs eight full design rounds.
Handbooks 01 / September 2026 / Premium PDF
The LLM Inference Systems Design Handbook
From one GPU to a fleet: how serving systems batch, cache, schedule and scale, with every number worked.
Follow one request from the socket to the last token. Size the KV cache for a 70B model on eight H100s. Choose an operating point from a latency target, then scale out, route, autoscale and debug the fleet. Seventeen chapters, one running example, and the arithmetic an interviewer expects you to do out loud.
The complete PDF and new editions are included while your Premium membership is active. Keep the copies you download.
Already a member? Sign in to download your copy →
58illustrated pages, 17 chapters
17diagrams and charts drawn from the worked numbers
8 + 42mock design rounds and self-check questions with answers
Read three pages before you decide
See how each chapter works through the numbers.
These are real pages from the PDF: the memory budget that sets concurrency on an eight-GPU node, the throughput-latency curve that picks an operating point, and the case for and against separating prefill from decode.
Who it is for
Written for the design round, and for the pager after it.
Moving from backend engineering into AI infrastructure
You know queues, caches and load balancers. The handbook shows where LLM serving breaks those instincts: memory that grows per token, steps that mix two workloads, and a tail that compounds.
Operating a serving fleet today
Failure drills start from what a dashboard shows and name the first metric to split. Admission, autoscaling and routing chapters give the signals to act on and the numbers to set them.
Already studying the concepts and the course
Concept pages explain one mechanism each. The handbook joins them: one running example carried through every decision, and self-check questions with worked answers at the end of each chapter.
Contents
Seventeen chapters, one system.
Foundations
- Follow one request from the socket to the last token
- Why decode waits on memory and prefill waits on arithmetic
- The KV cache decides how many users fit
The engine
- Batching, from fixed groups to every step
- Paging the cache, and what happens when it fills
- Reusing work that other requests already paid for
- Scheduling mixed work with a token budget
Service levels
- Measure what users feel: TTFT, ITL, tails and goodput
- Find the operating point, then size the fleet
Scaling out
- Splitting a model across GPUs for serving
- Separate prefill and decode only when it pays
- Making each decode step do more: speculation and precision
The fleet
- Admission, backpressure and autoscaling
- Many models, many tenants, one fleet
- Failure drills: symptoms, first checks and fixes
Practice
- Eight mock design rounds, with what a strong answer contains
Reference
- Formula sheet and glossary
A calculation from Chapter 3
How many users fit on eight H100s?
Llama 3.1 70B has 80 layers and 8 key/value heads of dimension 128. In BF16 each token's cache is 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, exactly 320 KiB. A 32,768-token conversation therefore holds 10 GiB of cache.
Give the engine 90 percent of each 80 GB GPU, subtract a 17.5 GB weight share and a 4 GB runtime reserve, and each GPU keeps 50.5 GB for cache: 404 GB across the node, about 1.23 million tokens. At an average of 4,096 tokens per sequence that is about 301 concurrent sequences. That is a memory ceiling. The handbook then shows why a 25 ms latency target lands nearer 130.
Try your own model in the KV-cache calculator → · Read the KV cache sizing concept →
Interview questions it prepares you for
Practice the prompts, then check the reasoning.
Each prompt below has its own worked answer on the site. The handbook gives the system-level reasoning that connects them.
- You have eight H100s and Llama 3.1 70B. How many concurrent users can you serve? →
- Design an LLM batching system end to end →
- Your p99 TTFT tripled and p50 did not move. Find the cause. →
- When does splitting prefill and decode onto separate pools pay? →
- When does speculative decoding help, and when does it break even? →
- Autoscale GPU replicas on queue depth without thrashing →
- Design serving for 100 fine-tuned variants of one base model →
- What does it cost per million output tokens on eight H100s? →
How it was made
Every number is derived, and every claim has a source.
Mechanisms come from the original papers, including Orca, PagedAttention, SGLang, Sarathi-Serve, DistServe, Splitwise, Mooncake and speculative decoding, and from vendor specifications and maintainer documentation. The handbook cites 35 primary sources by number. Its worked examples and self-check answers are recomputed by the build and must match the text before an edition ships.
Every chapter opens with links to the concept pages and questions that go deeper:
The KV cacheContinuous batchingPagedAttentionPrefix cachingChunked prefillTTFT, TPOT and goodputDisaggregated prefill and decodeSpeculative decodingMulti-LoRA serving
Reviewed 2026-09-24. Calculations are first-order estimates with stated assumptions; no GPU benchmark was run. Prices in examples are illustrative, not quotes. The cover is conceptual vector artwork.
Questions
Before you download
How do I get the PDF?
The complete PDF is included with active Premium membership. Sign in, then download it from this page. You keep the copies you download, and new editions are available while your access is active. The three sample pages above are free to read.
Does it replace the concept pages and the Inference Engineering course?
No. Concept pages explain one mechanism each, and the course teaches in short lessons. The handbook connects them into whole designs: one running example carried from a single request to a sized, routed and debugged fleet, with the arithmetic done at every step. Every chapter links to the concept pages and questions that go deeper.
Which engines and hardware does it cover?
The mechanisms apply to any modern serving engine. Examples cite vLLM and SGLang documentation, NVIDIA Dynamo, llm-d and the Kubernetes Gateway API Inference Extension, and the worked numbers use H100 specifications. Engine flags change between releases, so the handbook names the setting and links the maintainer's page.
Are the numbers benchmarks?
No. They are first-order calculations from published specifications and model configurations, with every assumption stated so you can replace it with a measurement. Where a paper reports a measured result, the handbook attributes it to the authors. No GPU benchmark was run for this edition.
Are the interview rubrics from real companies?
No. The rubrics in the mock rounds describe what strong answers tend to contain. They are a study aid written for this handbook, not any company's grading criteria.
Keep the other guides beside it.
The AI Infrastructure Field Guide builds the foundations from the first tensor to distributed systems. The AI Accelerator Field Guide compares GPUs and TPUs and is free with an account. The Open Model Field Guide covers model releases, architectures and deployment paths.