mediumNewTogether AIAnyscale

Draw the throughput versus latency curve for LLM serving and explain where you would operate on it.

Tokens per second climbs with batch and per-token latency climbs with it. The curve has a knee set by the hardware ridge, and the operating point is where your latency SLO cuts it.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Tokens per second climbs with batch and per-token latency climbs with it. The curve has a knee set by the hardware ridge, and the operating point is where your latency SLO cuts it.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputAn LLM request has two latencies, not one: time to first token, set by queueing and prefill, and time per output token, set by the decode loop. Reporting them as percentiles, and reporting goodput (requests that met both SLOs per second) rather than raw throughput, is what separates a serving engineer from a benchmark reader. The numbers a loop expects: about 24 tokens per second single-stream for a 70B model on one H100, TTFT floors in the hundreds of milliseconds for long prompts, and p99s that come from queueing, not from the GPU.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Foundational

🧩 GPU & Accelerator Architecture

Roofline ModelThe roofline plots a kernel's attainable throughput against its arithmetic intensity, FLOPs per byte moved from memory. Below the ridge point (peak FLOPS divided by memory bandwidth, about 295 on an H100 in bf16) a kernel is memory-bound and no amount of clever code reaches the peak; above it, compute is the limit. One picture explains why decode runs at under 1% of peak and why fusion and batching are the two levers that move it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the fastest serving path for a 70B model with a 200 ms TTFT target at production scale.Next in this tracknvidia-smi shows 30 percent utilization on our serving fleet. Is that a problem, and what would you look at instead?Next in this trackWhat changes when you serve a vision-language model instead of a text-only one?

FEDITOR'S NOTE

Scored on deriving the curve from the bandwidth formula rather than sketching it, on locating the knee at the ridge point, and on placing the operating point with an SLO and a cost number.

DISCUSSION · 0

No comments yet — be the first to share your approach.