Serving Benchmarks That Do Not Lie
Design reproducible serving experiments with realistic arrivals, cache state and failure accounting. Interpret latency-qualified throughput and conditional per-rank bounds without turning optimistic kernel estimates into production capacity.
TL;DR: A serving benchmark should answer a stated decision with a reproducible workload, arrival process and measurement boundary. Report offered load, failures and latency-qualified completions together. Separate cold-start behavior from warmed operation, and preserve prompt/output correlations and prefix reuse. Use per-rank traffic and compute estimates to investigate performance, but do not treat active parameter bytes divided into summed GPU bandwidth as a universal end-to-end ceiling. Capacity is the tested operating region that satisfies the service contract.
Define what the number is for
A typical situation: a new checkpoint produces more output tokens/s than the old one, but generates longer answers and has worse tool-call success. That token rate alone cannot decide the migration. State the task-quality threshold, latency and error contract, and resource budget first.
For an engine comparison, initially hold model, precision, hardware and workload fixed. For a deployment comparison, allow documented configurations to differ while holding the product contract and cost accounting fixed. Both experiments are useful; they answer different questions. Reading config.json to Size a Model You Have Never Run explains why model identity includes architecture and runtime state, not only parameter count.
Preserve the workload that causes the bottleneck
| Dimension | Record | Boundary case to test separately |
|---|---|---|
| Inputs and outputs | Joint length distribution, task mix, tokenizer/template | Maximum accepted context, long generations |
| Arrival process | Actual timestamps, bursts, admitted load | Overload and recovery |
| Cache state | Hit lengths, reuse distance, routing, eviction | Cold replica, working set larger than cache |
| Execution | Engine/image, kernels, dtype, TP/EP/PP, resolved flags | Supported alternative layout |
| API behavior | Streaming, parser, cancellation, stop behavior | Invalid output or interrupted stream |
Sampling prompt and output lengths independently can lose correlations: long documents may systematically request long answers. Preserve conversation order and realistic delays between turns too. A constant-length test is a valid controlled experiment; it simply does not represent a variable production mix by itself. It can still have latency variation from queueing, scheduling and hardware behavior.
Separate a fixed-concurrency diagnostic from independently scheduled arrivals. With fixed concurrency, a slower response delays the next request, reducing offered load. In the vLLM 0.28.0 benchmark CLI, request rate and maximum concurrency are different controls, and the latter can reduce the actual arrival rate. Save achieved client and server arrival rates alongside the requested setting.
Count requests that disappear from a latency plot
Use a synthetic 100-second steady observation window, with no net change in unfinished requests. There are 3,000 offered requests: 100 rejected, 2,900 completed. Of the completions, 2,550 satisfy the full latency and correctness contract.
Offered rate = 3,000 / 100 = 30 requests/s
Completion rate = 2,900 / 100 = 29 requests/s
Qualified completion rate = 2,550 / 100 = 25.5 requests/s
Qualified fraction of offered requests = 2,550 / 3,000 = 85%
Completion-only qualification = 2,550 / 2,900 = 87.9%
The latter percentage excludes rejected requests. Neither proves a 99% success contract. A latency percentile computed only over completions must carry that denominator. Include timeouts, unfinished requests and retries; if retry attempts and logical user requests are both counted, report both populations. Latency Metrics: TTFT, TPOT and Goodput explains the timing boundaries.
Requests in a synthetic 100-second window
2,550: 85% of offered
350
100: include them
Specify whether goodput is requests/s, output tokens/s or completed tasks/s. It also needs a qualification rule: satisfying TTFT alone is different from satisfying TTFT, TPOT and task validity together. Two separate metrics each passing on 99% of requests guarantee only 98% joint passage in the worst case; measure the intersection.
Make bounds conditional and local
NVIDIA's performance guide separates memory, compute and latency limitations. For a specified step, a memory-time lower estimate is max over ranks(bytes accessed on rank / rank bandwidth). The traffic must include relevant weights, cache and other transfers at the memory level being modeled. A separate compute estimate uses operations and supported execution throughput.
Synthetic example: four ranks each have a hypothetical 2 TB/s memory ceiling. Their required HBM traffic for a synchronized step is 20, 20, 20 and 40 GB:
Rank memory-time floors = 10, 10, 10 and 20 ms
Step memory floor = max(10, 10, 10, 20) = 20 ms
Incorrect aggregate shortcut = 100 GB / 8 TB/s = 12.5 ms
The overloaded rank cannot borrow the other ranks' unused bandwidth. Compute dependencies and exposed communication can make the real step slower. Summed bandwidth is a useful idealization only when work is suitably distributed and concurrent.
Active parameters are not a direct HBM-traffic counter. Batch reuse, expert unions, mixed formats, cache residency and repeated reads change bytes moved. Speculative verification changes both step work and accepted tokens per step. Compare the right numerator with the same execution regime; a batch-one weight-only estimate cannot diagnose batched aggregate token/s by simple division. A gap from an optimistic bound is a hypothesis, not proof of recoverable headroom.
Report enough evidence to make the decision
Plot offered rate against tail latency, qualified throughput, rejection and backlog growth. Test around the acceptance boundary, then leave margin for bursts and drift. Peak throughput may be appropriate for offline work; it is unsuitable for an interactive choice when it violates the contract.
Report warmup policy, cold-start results, duration, sample counts and repeated-run variation. At 1,000 samples, only about 10 observations occupy the upper 1% of a continuous distribution; 10,000 gives about 100. That is intuition about tail evidence, not a guarantee of a stable p99. Correlated requests and nonstationary load require longer or repeated windows. Do not average per-replica percentiles and label the result the fleet percentile.
Interviewers are listening for a decision and its limits. Choose the largest tested workload that meets the whole contract with justified margin; reverse the recommendation if cold replicas, long outputs or recovery invalidate it. Save the trace definition, raw results and exact artifact manifest so someone can rerun the comparison.
Key takeaways
- In the synthetic window, 29 completions/s becomes 25.5 qualified completions/s and only 85% of offered requests qualify.
- Per-rank traffic gives a 20 ms memory floor where a summed-bandwidth shortcut gives an invalid 12.5 ms estimate.
- Separate fixed concurrency, independent arrivals and cold-start experiments.
- Tail percentiles need sample counts, failure accounting and repeated evidence; the headline rate is only one result.