Define TTFT, TPOT and goodput, and tell me how you would measure each one in production.

TL;DR: Choose one measurement boundary. TTFT runs from request start to first output; per-request TPOT averages the n−1 later token gaps, while ITL retains each gap. Count goodput from valid requests meeting the complete latency contract per second. Keep offered load, failures and the observation window visible so a faster-looking dashboard cannot hide worse service.

How to approach it

Ask whether the product cares about first response, smooth streaming, a completion deadline or all three. Define the clock and token-counting convention, work one timestamp example, then explain how to retain failures and evaluate both latency limits on the same request. Finish with the evidence needed before tuning or scaling.

A strong answer

Suppose throughput rises after increasing the batch size, but users report pauses. Latency Metrics: TTFT, TPOT and Goodput gives the measurements needed to investigate those pauses. A count of tokens produced across all requests cannot tell you how long a particular person waited.

Use client receipt times for this answer. Let request start be 0 ms and actual token receipts be 100, 120, 320 and 340 ms. TTFT is 100 ms; individual inter-token gaps are 20, 200 and 20 ms; mean TPOT is 80 ms. The average is below a 100 ms mean-TPOT threshold, yet the stream contains a 200 ms pause. A no-long-pause requirement needs its own gap or per-request maximum-gap condition.

The long blank interval in the timeline is the pause a person feels. The average pace summarizes the whole answer and loses that location.

This small Python example computes those observations. Inputs are token timestamps relative to request start, in milliseconds, from one monotonic clock (one that never jumps backward, unlike wall-clock time after a time sync); it rejects out-of-order or negative times.

from math import isfinite

def token_metrics(times_ms):
    if any(not isfinite(t) or t < 0 for t in times_ms):
        raise ValueError("receipt times must be finite and nonnegative")
    if any(b < a for a, b in zip(times_ms, times_ms[1:])):
        raise ValueError("receipt times must be ordered")
    if not times_ms:
        return None, None, []
    gaps = [b - a for a, b in zip(times_ms, times_ms[1:])]
    tpot = sum(gaps) / len(gaps) if gaps else None
    return times_ms[0], tpot, gaps

assert token_metrics([100, 120, 320, 340]) == (100, 80, [20, 200, 20])
assert token_metrics([100]) == (100, None, [])
assert token_metrics([]) == (None, None, [])

Sanity check: last-token latency is TTFT + (n − 1) × TPOT = 100 + 3 × 80 = 340 ms. This endpoint excludes any later completion-control event. For one output token, TPOT is undefined. Empty output must be handled by the success/failure contract, not inserted into latency histograms as zero.

At a streaming API boundary, a chunk may contain multiple tokens. Empty role or control events should not start the first-output clock. If exact token receipt times are unavailable, report chunk gaps honestly and obtain a reliable output-token count for any request-level pace estimate. Never silently label transport chunks as individual tokens.

Goodput requires a joint decision. For a hypothetical twenty-second offered-arrival cohort of 100 requests, assume all return valid multi-token responses by their deadline. Ninety meet TTFT ≤ 500 ms and ninety meet mean TPOT ≤ 50 ms. If the failures are disjoint, only eighty meet both:

Compute the rate and the fraction with different denominators:

offered cohort = 100 requests over 20 seconds
valid requests passing both limits = 80
goodput = 80 / 20 seconds = 4 requests/s
attainment = 80 / 100 offered requests = 80%

The 100 requests belong to the arrival window; finalize their outcomes after completion or deadline. Failures, rejected offered requests and timeouts cannot become good requests, and they stay in the attainment denominator. Define how client cancellations, retries and legitimate one-token outputs are classified before running the test. A retry must not turn one user operation into two successes.

The DistServe paper motivates optimizing capacity subject to latency constraints. Be precise when comparing results: observed passing requests per second and maximum sustainable offered rate at a target attainment are related but different measurements.

Instrument request start, first/last output, output count, outcome, and individual chunk or token times at the client. Correlate request IDs with server queue, prefill, decode and emission events; compute durations within each clock domain rather than subtracting unsynchronized machine clocks.

The vLLM production metrics reference distinguishes these server histograms:

Use the deployed version's reference for exact metric names and definitions. Client buffering can create different observed gaps. Report percentiles with their window, count and population; merge compatible histogram buckets across replicas rather than averaging their p99 values. Include an offered-load sweep and representative lengths/cache states to expose where queues grow.

I would accept a throughput increase only if the target workload still meets its validity and latency contract. I would change the operating point if joint attainment drops or queue age grows. Do not autoscale on goodput alone: a demand drop lowers goodput without an overload, and completion-based signals can arrive too late. Capacity and Backpressure connects the demand and queue signals to that decision.

What interviewers probe next

• “Is p99 TPOT the same as p99 ITL?” No. One samples a mean per request; the other samples individual gaps, giving long responses more observations. Add a per-request maximum-gap metric if that is the product's concern.
• “Do two passing p90 charts prove 90% joint attainment?” No. The example gives 80% when their failures are disjoint. Evaluate the conjunction on each request.
• “Why did tail TTFT rise?” Inspect slow traces for queueing, cache misses, execution and delivery delays. The percentile alone does not identify the cause.

Common mistakes

• Starting the clock after admission and silently excluding the queue.
• Multiplying TPOT by n rather than n−1 when reconstructing last-token latency.
• Treating a one-token request as zero TPOT, or excluding failed requests from attainment.
• Calling goodput a percentage or averaging replica percentiles.

Key takeaways

• The four-token example has 100 ms TTFT, 80 ms mean TPOT and one 200 ms gap.
• Ninety of 100 passing each individual limit can mean only eighty pass both.
• Eighty good responses from a twenty-second offered cohort give 4 good requests/s and 80% attainment.
• Specify measurement boundaries and failure handling; use offered demand, queues and joint outcomes to judge a tuning change.