Design the benchmark you would run before putting a new open-weights model into production.

TL;DR: Define the deployment decision and acceptance contract, pin the artifacts, then test representative traffic across offered load. Preserve prompt/output correlations, prefix reuse and arrival timing. Report latency distributions, completions meeting the contract, rejected or unfinished requests, and recovery behavior together. Use controlled microbenchmarks and per-rank bounds to explain bottlenecks, not to replace the service test. Recommend a measured operating region with margin; do not interpolate an untested capacity from a peak token/s number.

How to approach it

First establish quality, API, latency and failure requirements. Then define a reproducible workload and resource budget. Run correctness smoke tests, controlled diagnostics and the service load sweep in that order. Close with the acceptance result, uncertainty, and the artifacts another engineer needs to reproduce it.

A strong answer

A team recommends a new open-weight model after a local test reaches 8,400 output tokens/s. The report omits offered arrivals, input lengths, cache state and errors. The number could be perfectly measured and still be insufficient to justify production use.

I would write the acceptance contract before running the benchmark. For this synthetic example, require at least 99% of offered logical requests to complete correctly within 1,000 ms TTFT and 50 ms per-request TPOT, with a separate limit on individual streaming gaps. Define timeout, cancellation and invalid-tool-output handling too. A task-quality evaluation must pass independently; a fast but incorrect model does not qualify. Serving Benchmarks That Do Not Lie separates the benchmark's populations and assumptions.

Build a workload manifest

If production data is unavailable, define several labeled synthetic distributions and boundary cases. Do not claim they predict production p99. Fixed-length prompts are useful for isolating a kernel or scheduler behavior; they still produce latency distributions, but omit variation in workload shape. Protect private prompt content while retaining the properties required by the experiment.

Separate three test families

First, validate the serving contract: chat formatting, stop behavior, tool and structured-output parsing, context boundaries, cancellation and quality fixtures. Next, use small controlled tests for cold loading, single-stream decode and fixed-shape batches. Finally, replay the intended service mix at multiple independent arrival rates, including bursts and overload. Keep cold-replica behavior as its own result rather than deleting it as warmup.

The vLLM 0.28.0 benchmark CLI distinguishes requested arrival rate from a client concurrency cap. A cap can throttle actual arrivals when responses slow. Measure when the client intended to send, when it actually sent, and when the server received each request. Otherwise a claimed 40 requests/s test may never offer 40 requests/s to the server.

Calculate the decision metric

Use the following invented steady-window results, each over 100 seconds with no net change in unfinished work. All completed requests are included in the reported completion rate; qualification adds the full contract. Rejections are the remaining offered requests in this simplified example.

At 30 requests/s for 100 seconds:
  offered = 3,000; completed = 2,900; qualified = 2,550; rejected = 100
  qualified throughput = 2,550 / 100 = 25.5 requests/s
  qualified fraction = 2,550 / 3,000 = 85%
  completed but outside contract = 2,900 - 2,550 = 350
sanity: 2,550 + 350 + 100 = all 3,000 offered requests.

The 30 requests/s run has higher qualified throughput than the 20 requests/s run, yet fails the 99% contract. Optimizing goodput alone is insufficient when the product also promises an attainment fraction. Only the 10 and 20 requests/s points pass the illustrated joint criterion; other acceptance tests remain. Exactly 99% observed at 20 requests/s leaves no statistical or operational margin. Repeat below and around that boundary before choosing production load. No 25 requests/s capacity has been measured here.

Latency Metrics: TTFT, TPOT and Goodput explains why average token rate cannot establish a tail target. Report per-request TPOT and individual token-gap tails separately. Count rejected, timed-out and unfinished work; do not let a shorter observation window make slow requests disappear. State how boundary-crossing requests enter the cohort and how the run drains.

Investigate performance without overclaiming a bound

A slow operating point deserves a trace of host scheduling, kernels, collectives and memory traffic. Estimate the work per rank for the actual batch and context. NVIDIA's performance guide explains why low arithmetic intensity does not guarantee saturated memory bandwidth when work is too small or dependency-limited.

For a simple synchronized step, the slowest rank's bytes/bandwidth estimate is more informative than total bytes divided by all GPUs' bandwidth. Active parameter count does not establish actual HBM reads, and a batch-one weight-only bound cannot be compared directly with batched aggregate token/s. A low fraction of an optimistic bound tells us to investigate; it does not prove a particular kernel change will recover the gap.

Publish a reproducible recommendation

Save the artifact manifest, workload definition, resolved flags, raw request outcomes, time boundaries and repeated-run results. Report sample counts: 1,000 requests provide only about 10 observations in the upper 1%, before accounting for temporal correlation. More samples and independent repetitions improve evidence but do not guarantee that rare production conditions have been covered. Avoid averaging percentiles across replicas.

I would recommend only a tested region that passes the complete contract with justified margin. Withdraw that recommendation if a representative quality, cold-start, long-output or recovery test fails. For a pure engine experiment, isolate one change first; for a deployable comparison, permit documented tuning while keeping the product and resource constraints fixed. No GPU benchmark was executed for the synthetic results above.

What interviewers probe next

• "Why not use only p99 TTFT?" It misses slow generation, incorrect results and rejected requests. Measure the full contract and its denominator.
• "How long should a run last?" Long enough to exercise relevant cache reuse, thermal and traffic timescales, with enough tail samples and repeated trials. State both duration and sample count.
• "Can different quantization be compared?" Yes as competing deployments if both pass quality and the format difference is explicit. It does not isolate an engine implementation change.
• "Is a peak throughput result useless?" No. It can answer an offline throughput question or a controlled diagnostic; it cannot establish an interactive operating point alone.

Common mistakes

• Reporting target arrival rate without checking achieved arrivals.
• Dropping unfinished requests or retries from the accounting.
• Inventing an untested operating point by interpolating a sparse sweep.
• Treating a model-card FLOP or parameter figure as measured runtime traffic.

Key takeaways

• At 30 offered requests/s, 25.5 qualified requests/s is only 85% attainment and fails the example's 99% contract.
• The 20 requests/s point is a boundary candidate, not a production guarantee.
• Preserve arrival timing and joint workload structure, and report all request outcomes.
• Use traces and conditional bounds for diagnosis; service capacity still requires a service experiment.