Latency Metrics: TTFT, TPOT and Goodput
An LLM request has two latencies, not one: time to first token, set by queueing and prefill, and time per output token, set by the decode loop. Reporting them as percentiles, and reporting goodput (requests that met both SLOs per second) rather than raw throughput, is what separates a serving engineer from a benchmark reader. The numbers a loop expects: about 24 tokens per second single-stream for a 70B model on one H100, TTFT floors in the hundreds of milliseconds for long prompts, and p99s that come from queueing, not from the GPU.
TL;DR: TTFT = queue wait + prefill time (+ scheduling and network). TPOT (also called inter-token latency) = the decode step time, which is bytes per step over bandwidth plus batch effects. End-to-end latency = TTFT + TPOT × output tokens. Throughput is tokens per second across the fleet; goodput is requests per second that met both a TTFT and a TPOT SLO, and it is the number to optimize because throughput can rise while every user's experience gets worse. Report p50 and p99 of each, separately, because their causes differ.
The two latencies and where they come from
A request's timeline has a shape, and each segment has an owner.
| Segment | What sets it | Typical scale (70B on TP8 H100) |
|---|---|---|
| Queue wait | load and admission policy | 0 at low load; unbounded near saturation |
| Prefill | prompt tokens × 2N FLOPs ÷ achieved FLOPS | ~40 ms for 1k tokens, ~300 ms for 8k |
| First token out | prefill end plus one decode step | adds one TPOT |
| Each decode step | weights + KV bytes ÷ bandwidth, plus batch and chunking effects | 20 to 60 ms per token depending on batch |
| Stream to client | network and serialization | single-digit ms per token |
TTFT is everything up to the first token, and it is dominated by prefill for long prompts and by queueing under load. TPOT is the steady-state decode step and is dominated by the memory-bound weight read, so it is the number that quantization, batching and the hardware's bandwidth move. A 70B bf16 model on one H100 cannot beat about 42 ms per step single-stream (141 GB at 3.35 TB/s), which is the 24 tokens per second figure; spread over TP8 the weights per GPU shrink and the step drops, at the cost of per-layer all-reduces.
End-to-end latency is TTFT plus TPOT times the number of output tokens, so a 500-token answer at 40 ms per token takes 20 seconds regardless of how fast the first token appeared. Users perceive the two differently: TTFT is "did it start", TPOT is "can I read faster than it writes" (about 10 tokens per second is reading speed for English prose, so TPOT under 100 ms is comfortable for chat and far too slow for an agent chaining 20 calls).
Percentiles, and why the tail has different causes
The median and the tail of each metric come from different mechanisms, so they are reported separately and debugged separately.
Median TPOT is physics: the decode step at the replica's typical batch. Tail TPOT is interference: a long prefill landing on the replica (chunked prefill is the fix), a garbage-collection or allocator stall, a KV eviction and recompute, a straggler GPU in a tensor-parallel group.
Median TTFT is prefill time for the typical prompt. Tail TTFT is queueing: near saturation, arrivals wait for a slot, and wait time grows without bound as utilization approaches 100%. A fleet at 90% utilization can have a p99 TTFT ten times its p50 while the GPU is doing nothing wrong. This is the single most common cause of "p99 spiked while p50 stayed flat", and the fix is capacity or admission control, not a faster kernel (Capacity and Backpressure).
Throughput versus goodput
Throughput is tokens per second, summed over the fleet. It rises with batch size until the ridge point, and the easiest way to raise it is to admit more sequences per replica. Doing so raises every sequence's TPOT and, past a point, its TTFT. A fleet tuned for throughput can be one where no request meets its latency target.
Goodput, from the DistServe paper's framing, is requests per second served within both SLOs (for example, TTFT under 500 ms at p99 and TPOT under 50 ms at p99). It is the metric that couples the business's latency promise to the fleet's capacity, and it is what disaggregation, chunked prefill budgets and admission control are tuned against. Two replicas with the same throughput can differ by 3x in goodput. When an interviewer says "design for 10k requests per second", the implicit question is "at what SLOs", and goodput is the honest unit of the answer.
Measuring it properly
Client-side timestamps, not server-side, define the user's experience; measure TTFT from request send to first token received. Use realistic prompt and output length distributions, because TTFT scales with input and end-to-end with output, and a benchmark at fixed lengths hides the tails. Run at the target concurrency, not batch 1, and report the percentiles at that load. Warm the prefix cache the way production would. And separate the fleet-level metrics (goodput, utilization, queue depth) from the per-replica ones (step time, batch size, KV occupancy), because the fix for a bad number lives in different places depending on which layer it appears in.
What interviewers are listening for
"Define TTFT and TPOT and tell me what sets each" is a warm-up, and the mark is for naming the mechanism behind each (prefill and queueing; the weight read) rather than expanding the acronyms. The follow-up in reserve is "p99 TPOT is 40x p50, what is going on?", where the strong answer lists interference causes and names chunked prefill, then asks about utilization before blaming the GPU. The other follow-up is goodput: "throughput went up after your change and users complained; why?" The answer that sounds right and fails is "we optimized tokens per second", which is exactly the thing that happened.
Key takeaways
- TTFT = queue + prefill; TPOT = the decode step (weights and KV bytes over bandwidth); end-to-end = TTFT + TPOT × output tokens.
- Report p50 and p99 separately: medians are physics, tails are interference (TPOT) and queueing (TTFT).
- Goodput, requests meeting both SLOs per second, is the metric to optimize; throughput can rise while every user gets worse.
- Single-stream 70B bf16 on an H100 is about 24 tokens per second; TP, quantization and batching move that number, queueing moves the tail.
- Measure client-side, at target concurrency, with realistic length distributions and a warm cache.