Serving Benchmarks That Do Not Lie

TL;DR: Fix four variables before you measure anything, then report a curve rather than a point. The input distribution, meaning prompt and output lengths drawn from your real traffic rather than from a fixed pair. The concurrency, because tokens per second at batch 1 and at batch 256 are different measurements of different things and both are legitimate. The cache state, because a benchmark that replays the same prompt with prefix caching on is measuring the cache. And the metric, since output tokens per second per user, aggregate tokens per second across the server, time to first token and end-to-end latency all answer different questions and are routinely confused. Then compute the bandwidth bound from the model's active parameter bytes and the hardware's memory bandwidth, and report your result as a fraction of it. Real deployments of large sparse models land around a tenth of that bound at batch 1, and knowing that is what turns a number into a diagnosis.

The bandwidth bound, and why to compute it first

decode is bandwidth-bound, so the ceiling is fixed by two numbers
  upper bound on tokens/s = (aggregate memory bandwidth) / (bytes of active weights read
                            per token)

worked for Kimi K3 on one 8-GPU B300 node
  active parameters: 104B, per the published model description
  most of those are expert weights in MXFP4 at about 0.53 bytes; assume 85% experts:
    104e9 x 0.85 x 0.53125 = 47 GB
    104e9 x 0.15 x 1.0     = 16 GB
    total per token        = 63 GB
  aggregate bandwidth: 8 GPUs x 8 TB/s = 64 TB/s
  bound = 64e12 / 63e9 = about 1,020 tokens/s at batch 1

what the project actually measured
  the vLLM launch note for this model reports 111 tokens/s at tensor parallel 8 and
    118 at tensor parallel 16, at batch size 1
  achieved fraction = 111 / 1,020 = about 11%
sanity: an 11% fraction is not a bug. At batch 1 the per-layer work is tiny relative to
        kernel launches, the expert all-to-all and sampling, and expert weights are read as
        many small scattered pieces rather than one stream, so the bound is far from tight

That gap is also the reason speculative decoding helps so much on these models: it amortizes the fixed per-step costs over several tokens. The same note reports 331 tokens/s at tensor parallel 8 with its speculative configuration, which the project describes as a 3.14 times improvement. Interpreting that as "the model got faster" is wrong; the per-step overhead got spread thinner, and the bandwidth bound did not move.

The four variables, and what happens when each is loose

The curve is the deliverable, not the peak. Peak aggregate throughput happens at a concurrency where per-user latency is unacceptable, so quoting it describes a configuration nobody would run. Latency Metrics: TTFT, TPOT and Goodput covers the metric definitions; goodput, meaning throughput counted only from requests that met their latency target, is the one that makes the curve collapse to a single honest number.

The ways a serving number turns out to be false

1. the cache was warm and the traffic will not be
   repeating one prompt with prefix caching enabled measures the radix tree
   fix: state the prefix-sharing rate and match it to production

2. the output length was short
   a benchmark generating 32 tokens is dominated by prefill and startup
   a product generating 800 tokens is dominated by decode
   these are different systems and the flags that optimize them differ
   fix: draw output lengths from production

3. the comparison ran different quantizations
   FP8 against bf16 is not an engine comparison, it is a format comparison
   fix: pin the format on both sides, and say which

4. concurrency differed between the two systems compared
   fix: sweep both and compare curves rather than points

5. the measurement included or excluded the queue
   end-to-end latency from the client includes queue wait; server-side TTFT may not
   fix: measure from the client, and report queue wait separately

6. a single run
   engines have warmup, autotuning and cache population; the first minute is not the steady
   state
   fix: discard a warmup window, run long enough for percentiles to be stable, report the
   interquartile range
sanity: a result that survives all six is usually a lot less impressive than the first
        number was, and it is the one that predicts production

A benchmark plan that fits on one page

• Traffic: prompt and output length distributions sampled from production logs, with the prefix-sharing rate matched.
• Sweep: concurrency from 1 up to well past the intended operating point, at least eight levels.
• Warmup: discard the first window, stated in the report.
• Metrics per level: TTFT p50 and p99, per-user output tokens per second, aggregate tokens per second, and goodput against the SLO.
• Context: engine and version, model and revision, quantization, GPU type and count, parallel degrees, and the flags that differ from defaults.
• The bound: the computed bandwidth ceiling, and the achieved fraction.

That last line is what makes a benchmark diagnostic rather than descriptive. A result at 11 percent of the bound at batch 1 and 60 percent at batch 64 tells you immediately where to look, and a result with no bound attached tells you nothing about whether more is available.

What interviewers are listening for

The bound computed first. It is a fast way to show that you understand what limits decode and it makes every subsequent number interpretable. After that, the four variables, and specifically the cache-state one, because benchmarking with prefix caching on and a repeated prompt is the most common way a serving number becomes meaningless. Interviewers also listen for the throughput-latency curve rather than a peak, since quoting peak throughput is the clearest sign someone has never had to hold an SLO.

Key takeaways

• Compute the bandwidth bound first: aggregate bandwidth divided by active weight bytes per token, which for Kimi K3 on 8 B300s is roughly 1,020 tokens/s at batch 1.
• Published measurements of 111 tokens/s at tensor parallel 8 put the achieved fraction near 11 percent, which is normal at batch 1 and explains why speculative decoding gives over three times.
• Fix four variables: input distribution, concurrency, cache state and which metric.
• Report a throughput-against-latency curve and the operating point where it crosses the SLO, not the peak.
• Every benchmark report should carry engine version, model revision, quantization, hardware, parallel degrees, non-default flags, and the achieved fraction of the bound.