Benchmarks lie unless the length distribution is real

TL;DR: Four things decide whether a serving benchmark predicts anything: a realistic length distribution for prompts and outputs, an arrival process rather than a fixed concurrency, latency reported at percentiles alongside throughput, and a run long enough to reach steady state. Miss the first and the rest barely matter.

Where you are. Second lesson of the module. You have the two components. This is how to measure them without producing a number that evaporates in production.

Why fixed lengths break the measurement

Send 1,000 identical requests of 512 prompt tokens and 128 output tokens. Every sequence enters together, uses the same cache, and finishes together. The engine looks excellent.

Nothing about that resembles production, and the difference is not a matter of degree.

Batch composition never varies. Real traffic mixes a 40-token prompt with a 40,000-token one, and the second changes what the scheduler can do for the first. A fixed-length run never exercises that interaction, which is where the interesting behaviour is.

Cache pressure is constant. Real cache occupancy moves as long sequences accumulate and finish. A fixed-length run finds a steady state immediately and stays there, so it never reaches the pressure where preemption starts.

The tail does not exist. With identical requests, p50 and p99 are the same number. The tail is generated by variance, so removing variance removes the thing you were trying to measure.

Prefill and decode never compete properly. The ratio between them is fixed, so the scheduler never faces the decision that dominates real latency.

The result is not a number with a known error bar. It is a measurement of a different system.

What a distribution looks like

Take it from production if you have it. If you do not, state what you assumed, because every number below rests on it and a reader cannot judge the result without it.

The mean is the wrong summary for all of these. Length distributions are typically skewed, and the tail is what saturates the cache and generates the p99. A benchmark tuned on the mean is tuned for a case that rarely occurs and never causes an incident.

Closed loop against open loop

This is the second most common benchmarking error and it is subtle.

Closed loop: a fixed number of workers, each sending a request and waiting for the response before sending the next. Concurrency is constant by construction.

Open loop: requests arrive at a rate, whether or not the previous ones have finished.

Closed-loop harnesses are the default in most tooling, and they hide overload. If the system slows, workers wait longer, so fewer requests are sent, so the offered load falls to match capacity. The system can never be overloaded, because the harness backs off for it. Real users do not.

That is why closed-loop benchmarks produce reassuring latency curves that flatten instead of exploding, and why the first real traffic spike behaves nothing like the test. If you want to know where the system breaks, you have to be able to offer more load than it can serve, and only an open loop can.

Report the pair, always

Throughput alone is meaningless because it can always be raised by batching harder until latency is unacceptable. Latency alone is meaningless because it can always be improved by serving less.

Report the curve: for each offered load, the throughput and the latency percentiles. The number that matters is the largest load at which both latency targets still hold, which is the goodput idea from Course 1 and the only figure worth sizing a fleet on.

Two more things that belong next to every result, because without them it is not reproducible: the length distribution used, and the duration. A run that stops before the cache reaches steady state measures the warm-up, and cache pressure builds over minutes, not seconds.

Do this before moving on

Write the benchmark specification for a service you know, before running anything: the prompt and output length distributions with their tails, whether they correlate, the shared-prefix fraction, the arrival process, the duration, and the two latency targets.

Then find the one you are least sure about. For most people it is the shared-prefix fraction, and it is also the one with the largest effect on the result, because it decides whether a large part of prefill happens at all.

Go deeper

• Latency Metrics: TTFT, TPOT and Goodput is the reporting contract these results have to satisfy.
• Capacity and Backpressure is what the open-loop region is actually measuring, and why the closed loop cannot reach it.
• Prefix Caching and KV Reuse is why the shared-prefix fraction moves the result more than almost anything else.
• Designing for Latency SLOs turns the resulting curve into the number you size on.
• Design a benchmark for a new model is this lesson as an interview question.
• Sanity check a claim of 10,000 tokens per second per GPU is the skill of disbelieving a headline, applied.

Key takeaways

• Fixed-length benchmarks remove batch variance, cache pressure and the tail, which are the things you were measuring.
• Capture distributions with their tails, not means; length distributions are skewed and the tail causes the incidents.
• Closed-loop harnesses back off when the system slows, so they cannot produce overload and cannot find where it breaks.
• Report throughput and latency percentiles together as a curve; the useful number is the load at which both targets still hold.
• State the distribution and the duration with every result, or it is not reproducible.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A benchmark harness runs 64 workers, each sending a request and waiting for the response before sending the next. Why does this understate overload behaviour?

   It does not exercise the tokeniser correctly64 workers is too few to saturate a modern acceleratorWaiting for responses adds client-side latency to the measurementIt is closed loop: when the system slows, workers wait longer and send fewer requests, so offered load falls to match capacity and the system can never be pushed past it

2. 2Why does using fixed-length prompts remove the phenomenon a serving benchmark is trying to measure?

   Answer it, then reveal

3. 3Two engines are compared and one reports 40% higher tokens per second. What single missing piece of information most undermines the comparison?

   The number of replicasThe latency at which each throughput was achieved, since throughput can always be raised by batching harder until latency is unacceptableThe hardware generation usedThe model's parameter count