Latency Metrics: TTFT, TPOT and Goodput
Read a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.
TL;DR: Time to first token (TTFT) measures the wait before an answer starts. Inter-token latency (ITL) measures each gap after that, and time per output token (TPOT) is the average gap for one request. With n output tokens, last-token latency is
TTFT + (n − 1) × TPOT. Goodput needs an explicit contract: here it counts valid requests meeting all latency limits per second, with failures and timeouts kept in the accounting.
Start with the timestamps
A model writes its answer in tokens: pieces of text, often a word or part of a word, produced one at a time and usually streamed to the user as they appear. (See Tokens, Embeddings and Logits for where they come from.) The first token takes longest, because the server must first read the whole prompt, a step called prefill. Each later token needs only one more decode step. Prefill vs Decode explains why the two phases cost different things.
An answer can start quickly and then freeze. Another can take a while to begin but stream smoothly. One total duration cannot tell those experiences apart. The first step is to choose where the clock lives: client-observed times describe delivery to the user; server times help explain the work behind it.
For this page, start the client clock when sending the request begins. Let t₁ be receipt of the first output token and tₙ receipt of the last. Ignore empty protocol events. For n ≥ 2:
| Metric | Calculation | What it tells you |
|---|---|---|
| Time to first token, TTFT | t₁ − request start | How long the initial wait lasted |
| Inter-token latency, ITL | tᵢ − tᵢ₋₁ for each later token | Whether the stream paused |
| Time per output token, TPOT | (tₙ − t₁) / (n − 1) | Average gap within this request |
A hypothetical request starts at 0 ms and receives four tokens at 100, 120, 320 and 340 ms:
TTFT is 100 ms. The three gaps are 20, 200 and 20 ms, so TPOT is (340 − 100) / 3 = 80 ms. Last-token latency is 100 + 3 × 80 = 340 ms. Sanity check: four output tokens have three gaps, not four. If a completion event arrives later, full response-completion latency includes that additional time; name which endpoint you measured.
The 80 ms average hides a 200 ms pause. Keep individual gaps, or at least a per-request maximum, when that pause matters. For one output token there is no later gap, so TPOT is undefined, not zero. A failed request with no token has no successful TTFT observation; its failure must still be counted.
A network event is not necessarily a token
Some streaming events contain several tokens, and some contain none. Speculative decoding can also release multiple tokens together. If the client sees only chunks of text, label their intervals chunk gaps; you cannot recover exact per-token arrival times by pretending one event equals one token. Count output tokens consistently with the model's tokenizer or authoritative usage data and document the measurement limitation.
The vLLM metrics reference exposes distinct histograms for inter-token latency and per-request time per output token. Check the deployed version and where it timestamps emissions. Those server observations need not equal client receipt times after buffering and network delivery.
Percentiles need a population
A percentile summarizes many measurements by rank. Sort 1,000 TTFT values from fastest to slowest: the p50 (median) is the 500th value, and the p99 is the 990th, so 1% of requests waited longer. Services set their latency promises, called service-level objectives (SLOs), on high percentiles like p95 or p99, because the average hides the slow requests users complain about.
A p99 of per-request TPOT gives each eligible request one observation. A p99 of all token gaps gives a long response many more observations than a short one. Both can be useful, but they answer different questions. Report the window, sample count, workload and aggregation rule; do not average per-replica p99s and call the result fleet p99.
A tail spike does not identify its cause. Inspect slow-request traces. Queueing, cache misses, long prefills, preemption, communication and delivery can all contribute. Prefill vs Decode and Chunked Prefill explain two mechanisms; a percentile alone cannot tell you which occurred.
Throughput is a different number
TTFT and TPOT describe what one user experiences. Throughput describes what the whole server delivers: total output tokens per second across every request it is running. A server streaming 64 requests at once, each with a TPOT of 25 ms, gives each user 1,000 ÷ 25 = 40 tokens per second, and delivers 64 × 40 = 2,560 tokens per second in aggregate. Raising the batch to 128 might lift aggregate throughput while pushing each user's TPOT up, so a faster server by one number can feel slower by the other. Always say which one you mean: per-request pace or aggregate throughput.
Count requests that satisfy the whole promise
Define a valid successful response and its deadlines before measuring goodput. In this example the limits are TTFT ≤ 500 ms and per-request mean TPOT ≤ 50 ms. All successful responses have at least two tokens. Over a ten-second offered-arrival window, five requests have these final outcomes after waiting for completion or deadline:
| Request | TTFT / TPOT | Meets both limits? |
|---|---|---|
| A | 100 / 20 ms | Yes |
| B | 200 / 40 ms | Yes |
| C | 700 / 20 ms | No: first token late |
| D | 100 / 80 ms | No: generation too slow |
| E | Timed out | No: failed response |
Observed goodput for this offered cohort is 2 / 10 = 0.2 requests/s. SLO attainment is 2 / 5 = 40%. They are different units. Keep the timeout in the offered-request denominator; excluding it would flatter the service. In a live dashboard, label a cohort provisional until outcomes are known, or define a completion-window convention and report unfinished work separately.
This is one explicit goodput convention. Capacity papers can instead ask for the maximum offered rate that maintains a target attainment percentage; DistServe uses this kind of latency-constrained capacity framing. State the convention rather than comparing unlike numbers.
Turn the measurements into a decision
Tune for more valid, timely completions at a representative offered load. Increasing batch size may increase token throughput while reducing the fraction meeting latency limits. Conversely, observed goodput can fall simply because fewer requests arrived. Read demand, queue age, failures and latency together before scaling; goodput alone is not a reliable leading autoscaling signal.
For interactive chat, keep initial wait and stream continuity visible. A batch workflow may tolerate slower starts but still have a completion deadline and cost target. Choose the contract for the product instead of assuming one latency target fits every application.
Check your understanding
Ninety of 100 requests pass the TTFT limit, and ninety pass the TPOT limit. Must ninety pass both? No. If ten fail only TTFT and a different ten fail only TPOT, eighty pass both. Evaluate the joint pass/fail condition on each request; separate percentile charts do not establish joint attainment.
Key takeaways
- Four output tokens have three later gaps; in the trace TTFT is 100 ms, TPOT 80 ms and last-token latency 340 ms.
- Mean TPOT can hide a 200 ms pause. Keep individual-gap information when continuity matters.
- Declare clock boundaries, token versus chunk observations and percentile populations.
- Per-request pace and aggregate throughput differ: 64 streams at 25 ms TPOT give 40 tokens/s each and 2,560 tokens/s in total.
- Goodput has units of requests per second; attainment is a fraction. Count valid responses meeting the whole contract, including failures in the accounting.