The LLM Inference Systems Design Handbook

What the diagram shows

GPU memory budget, in decimal GB. Of 80 GB, 8 GB is outside the engine allocation. Within its 72 GB allocation, 17.5 GB holds weights, 4 GB is runtime reserve and 50.5 GB is KV cache. Eight TP8 GPUs provide 404 GB of logical cache. At 320 KiB per token, the memory ceilings are 602 sequences at 2,048 tokens each, 301 at 4,096, 150 at 8,192 and 37 at 32,768. These counts do not guarantee a latency target.

One GPU of a TP8 node under the stated assumptions: 0.9 memory fraction, 17.5 GB weight share and a 4 GB runtime reserve. The concurrency bars are memory ceilings, not latency guarantees.

Three cautions keep this answer honest. First, "tokens held" is prompt plus output so far, and it grows during the response, so plan on the length at completion, not at arrival. Second, the table is a memory ceiling. Whether 301 sequences at 4K context also meet a latency target is a separate question that Chapter 9 answers, and usually the answer is no. Third, the reserve and utilization fraction are assumptions; the engine prints the real KV capacity at startup, and that printed number beats any napkin.

Three ways to buy more cache

Fewer bytes per element. Storing keys and values in FP8 halves the cache to 160 KiB per token and roughly doubles the sequences that fit. vLLM exposes this as --kv-cache-dtype.[26] Quality has to be measured on your workload, because attention can be sensitive to key precision in particular.

Fewer weight bytes per GPU. Every gigabyte of weights removed becomes cache. FP8 weights free about 8.75 GB per GPU in the example. Across the node that is 70 GB, room for about 214,000 more tokens of cache. Chapter 12 treats weight and KV precision as two separate decisions, because they are.

What the diagram shows

Decode-only throughput versus step time. At batch 1: 83 output tokens/s and 12.0 ms; batch 32: 2,112 and 15.2 ms; batch 64: 3,486 and 18.4 ms; batch 128: 5,168 and 24.8 ms; batch 256: 6,810 and 37.6 ms. A horizontal 25 ms target intersects near batch 130 and 5,200 tokens/s. The model assumes 4,096 cached tokens per sequence, 50 percent of peak bandwidth and 1.5 ms fixed overhead. Only batches up to the 301-sequence memory ceiling are plotted. It does not predict mixed-workload p99 latency.

Decode-only model: 4,096 tokens of context per sequence, 50 percent of peak bandwidth and 1.5 ms fixed overhead per step. The 25 ms line gives a candidate near 130 sequences. Mixed-load p99 latency still needs measurement.

Choosing the point

For a 25 ms ITL target, solve t(B) = 25 ms: about 130 sequences and about 5,200 tokens per second. Running 256 gives 31 percent more throughput but misses the step-time target. Running 64 gives about 33 percent less throughput, with more headroom for interference or jitter. Lower throughput alone does not prove that hardware is wasted.

Measure before trusting the model. Prefill shares the same steps, so a real mix yields fewer decode tokens per second, and the SLO is on p99 gaps, which chunked prefill, preemption and jitter push above the average. Run the load test from Chapter 8 and read the operating point off the measured goodput curve.[4]

Little's law turns traffic into concurrency

Little's law says the average number of items in a stable system equals their arrival rate times the average time each spends inside.[21] Choose the boundary first. Arrival to last token counts requests waiting, prefilling and decoding. The batch B in our curve counts only sequences actively decoding.

Worked example.

10 requests arrive per second and each takes 20 seconds from arrival to last token. On average 10 × 20 = 200 requests are in flight.

What the diagram shows

Time runs downward. Here the prefill pool sends token 1 before the KV handoff finishes. The remaining transfer delays token 2, so it affects the first token gap. An 8,192-token prompt carries 2.5 GiB. A design that withholds token 1 until handoff instead adds that wait to TTFT. The client sends a prompt to the prefill pool, which returns token 1. The cache handoff to the decode pool must finish before that pool produces token 2. The arrows show event order, not proportional elapsed time.

Time runs downward. Here the prefill pool sends token 1 before the KV handoff finishes. The remaining transfer delays token 2, so it affects the first token gap. An 8,192-token prompt carries 2.5 GiB. A design that withholds token 1 until handoff instead adds that wait to TTFT.

Transfer time.

Over one 400 Gb/s NIC at an assumed 50 GB/s effective: about 53.7 ms. Spread across eight such NICs, one per GPU: about 6.7 ms. Compare with about 290 ms to compute that prefill (Chapter 2).

Which latency grows depends on who sends the first token. In the figure, the prefill pool streams token 1 to the client before the handoff finishes. The remaining transfer delays token 2, so it appears in the first-to-second-token gap. If a design waits for the decode pool to send token 1, that remaining transfer instead contributes to TTFT. Trace the client-visible events before adding times.

Layer-by-layer transfer can overlap some of the copy with prefill, but the decode pool must have the required state before it uses it. The transfer needs a fast network and a library that moves GPU memory directly, such as NVIDIA's NIXL through vLLM's connector.[30] The 50 GB/s example is the raw 400 Gb/s rate used as an optimistic payload assumption; protocol and contention overhead make a real copy slower. Measure the effective payload rate and the portion hidden by overlap separately.

The other costs are operational. Two pools must be sized against each other, and the right ratio changes with the traffic's prompt-to-output mix. A burst of long prompts backs up the prefill pool while decode sits idle, or the reverse. Dynamo's planner and llm-d's scheduler exist largely to keep that ratio right as traffic shifts.[32][33]