A serving benchmark needs a workload and an arrival model

TL;DR: A benchmark is useful when its traffic matches the question you want to answer. Fixed request lengths help compare individual changes; a realistic mix shows how the service handles its users. Measure how much traffic the test actually sends, and report failures alongside speed so a faster result does not hide dropped requests.

Where you are. Second lesson of the engine module. One engine claims 40% more tokens/s. Before changing the deployment, check whether both runs used the same model, output quality, traffic distribution, cache state and latency target. A higher number can answer a different question.

Fixed lengths are useful but incomplete

A test where every request has 512 input tokens and 128 output tokens keeps the amount of work predictable. That makes it useful for comparing a kernel or changing one configuration setting. A service handling both short chats and long documents faces a different mix, so you need another test before predicting how it will behave.

Identical lengths do not imply identical latency. Requests can arrive at different times, queue behind different work, hit or miss a cache, or run on different devices. Even a simultaneous batch can have variable delivery and completion times. Fixed-length traffic can also fill the cache and trigger preemption; load level and admission policy still matter.

Use controlled tests to explain a mechanism, then validate the conclusion on representative traffic. If production really uses a narrow fixed shape, that distribution is relevant. Reverse a broad mixed-workload test when the question is a single kernel regression: it introduces variation that makes attribution harder.

Keep the relationships between requests' properties

Average lengths help estimate the total work arriving at the service. They do not tell you how much memory the longest requests need or how long those requests wait. Preserve the relationship between input and output lengths too: choosing them independently can create combinations that your users never send. If you cannot use production data, describe the synthetic traffic you chose and test how the result changes when uncertain inputs vary.

For generated outputs, report realized lengths as well as requested caps. Forcing every request to generate its maximum can be a useful stress test, but it is a different workload from natural stopping. Tokenization, chat templates and model revisions must be fixed or recorded.

Closed-loop tests apply backpressure

A closed-loop worker waits for its response before sending another request, possibly with a think time. With N continuously busy workers and mean response time R, achieved throughput is approximately N/R. Including mean think time Z gives N/(R+Z) under stable conditions.

32 workers, no think time:
mean response 0.5 s → 32 / 0.5 = 64 requests/s
mean response 2.0 s → 32 / 2.0 = 16 requests/s
sanity: response time quadruples while the harness request rate falls to one quarter

This test can still overload the service and create long queues. But when responses slow down, each test worker waits longer before sending again, so the test itself reduces the arrival rate. That represents a fixed group of users waiting for answers. It can miss how quickly a backlog grows when new requests keep arriving independently of earlier responses.

Open-loop tests need a capable generator

An open-loop test schedules arrivals independently of completions, using a constant rate, a stochastic process or recorded intervals. Suppose it offers 20 requests/s while a degraded service completes 16/s. With no rejection or cancellation, backlog grows by (20 − 16) × 30 = 120 requests over 30 seconds. A bounded queue instead forces explicit rejection or backpressure; measure those outcomes.

Record when the generator planned to send each request, when it actually sent it and when the response arrived. The generator may run out of CPU, connections or allowed concurrent requests and send late. A timer that starts only at the actual send time misses this delay. Measure that lag so you can distinguish a server handling the requested load from a generator failing to deliver it. The load-generator explanation of coordinated omission describes this measurement problem; the remedy must match the intended arrival model.

The inference load-mode reference distinguishes concurrency, request-rate and custom interval modes. Choose the mode deliberately and verify its realized load.

Audit the numerator as well as the timer

Suppose an optimization proposes four tokens at a time and lets the target model check them. A round emits its accepted draft prefix plus a correction or bonus, subject to stopping rules. Drafts that were rejected or generated past EOS are work performed, not useful output.

Use this invented two-round trace to audit the arithmetic. One stream runs back to back, with no EOS, output cap or idle gap. Durations include the full exposed draft, verification and bookkeeping work.

A: 5 / 0.010 = 500 tokens/s; B: 1 / 0.020 = 50 tokens/s
wrong combined rate = (500 + 50) / 2 = 275 tokens/s
actual combined rate = (5 + 1) / (0.010 + 0.020) = 200 tokens/s
sanity: ordinary decoding at 5 ms/token also emits 6 tokens in 30 ms

The mistaken average invents a 37.5% throughput gain. Weight the rates by their elapsed times, or divide total useful output by the observation duration. At concurrent load, use the fleet's shared wall-clock interval; summing overlapping request durations does not give aggregate throughput. Count completions and SLO-compliant goodput separately as well.

Now shorten round B from 20 to 14 ms, keeping the outputs unchanged. The rate becomes 6 / 0.024 = 250 tokens/s: 25% higher throughput from a 20% time reduction. Those percentages use different denominators. This trace proves the arithmetic only. Two rounds say nothing about stable p99, peak capacity or quality. Measure repeated representative runs, include proposal memory, and inspect user-visible gaps when several tokens arrive together. The Speculative Decoding concept derives the sampling rule and expected progress before this measurement step.

Report useful capacity and uncertainty

At each load level, report how many requests arrive, how many the service accepts and how many finish per second. Also report goodput: the work that meets the stated success and latency requirements. Show time to first token (TTFT), the gaps between tokens and end-to-end (E2E) completion time, alongside counts of failures and other outcomes.

A larger batch eventually stops improving throughput, and a longer queue may let the server accept requests that finish too late to be useful. Use the same workload and acceptance rules when comparing engines.

Separate cold startup, warm steady traffic and burst/recovery tests. Run long enough for the relevant effects, such as prefix-cache turnover, long generations or autoscaling, rather than prescribing an arbitrary universal duration. Repeat runs and report variation. A p99 from 100 completed requests has only about one observation in its upper 1%; it is weak evidence for a stable tail target.

Do this before moving on

Write a test matrix with one fixed-shape diagnostic, one representative concurrency sweep, one independent-arrival sweep and one burst/recovery run. Specify the cache initialization, output stopping policy and failure denominator. Then introduce 1 second of think time into the 32-worker example: at 2-second mean response time, throughput becomes 32/(2+1) ≈ 10.67 requests/s. State which user behavior that represents. For the two-round trace, write the numerator, time boundary and rejected-work counters you would collect; explain how that boundary changes for concurrent requests.

Go deeper

• Latency Metrics: TTFT, TPOT and Goodput is the reporting contract these results have to satisfy.
• Capacity and Backpressure is what the open-loop region is actually measuring, and why arrival-rate tests complement concurrency sweeps.
• Prefix Caching and KV Reuse explains why the same lengths can generate different amounts of prompt work.
• Designing for Latency SLOs turns the resulting curve into the number you size on.
• Design a benchmark for a new model is this lesson as an interview question.
• Sanity check a claim of 10,000 tokens per second per GPU is the skill of disbelieving a headline, applied.

Key takeaways

• Controlled fixed-shape tests and representative workload tests answer different questions.
• 32 closed-loop workers at 0.5 versus 2 seconds produce about 64 versus 16 requests/s without think time.
• An excess of 4 arrivals/s creates 120 queued requests in 30 seconds if nothing rejects or cancels them.
• Record realized arrivals and generator lag; configured load alone proves nothing about delivered demand.
• Divide total useful output by elapsed wall time: 6 tokens in 30 ms is 200 tokens/s, even when the two round rates average to 275. Report goodput and outcome counts separately.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 132 closed-loop workers have no think time. Mean response rises from 0.5 to 2 seconds. What happens to their approximate request rate?

   Answer it, then reveal

2. 2Do fixed prompt and output lengths guarantee equal p50 and p99 latency?

   No, but only because the tokenizer changesYes; only token-length variance creates latency tailsYes, if all requests use one modelNo; queueing, arrivals, cache state, scheduling and hardware can still vary

3. 3A rate test is configured for 20 requests/s, but the generator only sends 14/s. Is the 20/s capacity claim established?

   Yes, if server p99 is below the targetNo; report realized send rate and generator lag, remove the client bottleneck, then rerunYes; the configured rate is the workload by definitionNo; all rate-controlled tests are invalid

4. 4Two consecutive rounds emit 5 tokens in 10 ms and 1 token in 20 ms. What is their combined output rate?

   200 tokens/s: divide 6 tokens by 30 ms500 tokens/s: use the successful round400 tokens/s: divide 6 tokens by mean round duration275 tokens/s: average the two round rates

5. 5Round B drops from 20 to 14 ms, with the same outputs. What are the new combined rate and percentage throughput gain?

   Answer it, then reveal