TL;DR: First check what the theoretical bound counts and what the measured time includes. Active parameters are not a direct HBM-traffic count, and summed bandwidth assumes balanced concurrent work. Then profile the critical path: memory and compute kernels, exposed collectives, launch dependencies and host or output handling. Small local expert batches can underutilize hardware even with coalesced reads. CUDA graphs, kernel changes, parallel-layout changes, batching and speculation address different costs. A large gap neither proves misconfiguration nor establishes that it is normal or recoverable.
How to approach it
First pin the model, precision, context, group-wide batch and timing boundary. Recompute the candidate bound per rank. Then trace representative steps across the ranks and identify the longest dependency chain. Estimate the maximum gain from changing one component, then test it. Explain whether it improves the service and what result would make you undo it.
A strong answer
A spreadsheet predicts roughly 1,000 tokens/s, while a single stream produces about 100. Before tuning, check whether the spreadsheet counts only active weight payload and whether the observation includes scheduling, sampling, detokenization or transport. Comparing different boundaries creates a gap no kernel optimization can close.
Check the denominator
In a mixture-of-experts model, active parameters are the weights a token actually uses on a step; the rest sit in memory unused for that token.
Bandwidth-Bound Decode Throughput introduces the memory model. For this diagnosis, estimate actual required traffic at the memory level being modeled, on each rank. Include relevant KV reads and other state. Expert weights may be reused within a batch, accessed repeatedly by an implementation, or stored at mixed precision. A total or active parameter figure is only an input to that estimate.
Use a synthetic single-rank example with a 2 TB/s memory ceiling and 10 GB of required HBM traffic per ordinary decode step:
Memory-only time floor = 10 GB / 2 TB/s = 5 ms
One emitted token per step implies a memory-only ceiling of 200 tokens/s
If observed step time is 20 ms, observed rate = 50 tokens/s
Observed rate / optimistic ceiling = 25%
sanity: the floor accounts for 5 of 20 ms; it does not explain the other 15.
For multiple ranks, the maximum rank-local bytes/bandwidth term limits a synchronized step. Add compute and dependency constraints without double-counting overlap. A low fraction of the memory-only ceiling is not itself a diagnosis. NVIDIA's performance guide explains that small or insufficiently parallel work can be latency-limited despite a low arithmetic intensity.
Attribute the critical path
| Observation | Candidate explanation | Discriminating check |
|---|---|---|
| Small expert kernels dominate | Too little local work, inefficient shapes or packing | Kernel dimensions, occupancy and memory traffic at the same routing pattern |
| Gaps before ready kernels | Host launch overhead or scheduling | CPU/GPU timeline and a supported graph-replay comparison |
| Long collective intervals | Transfer cost or a late participant | Aligned rank timelines and prior computation |
| Device steps are fast, client gaps are long | Output processing, transport or buffering | Device completion versus server emission versus client receipt |
| Long-context slowdown | KV/indexer traffic or attention work | Same output workload at controlled context lengths |
Selected experts can reside in separate memory regions without making accesses inside their matrices uncoalesced. Do not blame random memory reads merely because routing is sparse. Similarly, sampling is not universally CPU work; inspect the implementation. TP can execute selected expert shards, so sparsity alone does not imply EP will fix the problem.
Use Nsight Systems for the cross-device and host timeline, then a suitable kernel profiler for a narrowed hypothesis. Sum only nonoverlapping critical-path components when constructing a step budget. Kernel-duration totals across ranks are not elapsed time. Serving Benchmarks That Do Not Lie explains the distinction between diagnostics and product capacity.
Bound one improvement before promising it
Here is a separate synthetic 9 ms critical path, with mutually exclusive terms: 4.5 ms device compute/memory work, 2 ms exposed communication, 1.5 ms launch dependencies and 1 ms other exposed work. Suppose a supported graph path reduces only launch dependencies to 0.5 ms:
Before = 4.5 + 2 + 1.5 + 1 = 9 ms
After = 4.5 + 2 + 0.5 + 1 = 8 ms
Single-stream speedup = 9 / 8 = 1.125×
sanity: removing 1 ms from 9 ms improves rate by 12.5%, not several times.
The vLLM CUDA-graph design describes supported capture and dispatch modes. Verify that the shapes in question actually replay a graph. Graphs have memory and compatibility costs and do not remove genuine data dependencies or all host-side work. Empty GPU time is a reason to inspect the timeline, not proof that replay will eliminate it.
Batching and speculation change the work being compared
With ordinary one-token decode for each of B active sequences, aggregate rate is approximately B / step time. Batching can reuse weight reads, but the expert union, KV traffic and compute can also grow. Step time need not remain fixed, and a larger batch does not guarantee a particular change in per-user latency. Test the service's concurrency regime; single-stream behavior can still matter during quiet periods or on a newly started replica.
Speculation can emit several tokens from one target verification cycle. Under a simplified constant conditional acceptance probability a, draft length k, and one correction or bonus token per cycle, expected emitted tokens are:
E[tokens/cycle] = 1 + a + a² + ... + a^k
= (1 - a^(k+1)) / (1 - a), for a != 1
At a = 0.8 and k = 7: E = 4.161 tokens/cycle
At a = 1: E = k + 1 = 8
That is a model of accepted-prefix lengths, not a universal formula for every decoder or an arbitrary average acceptance counter. The speculative-decoding paper explains the sampling construction. Speculative Decoding covers the mechanism.
Suppose the synthetic verification cycle costs 12 ms in the target, 3 ms drafting and 1 ms other work, with no overlap. Against 9 ms ordinary decode, speedup is 4.161 × 9 / 16 = 2.34×. If a falls to 0.2 with the same costs, expected output is about 1.25 tokens and speedup falls to 0.70×, a slowdown. Measure accepted output, draft cost, verification cost and extra memory together. Fewer target invocations alone does not establish a win.
I would fix a demonstrated launch or kernel problem first, then compare supported layouts and speculation at the intended workload. Undo a change that improves an isolated metric but worsens the service's latency, quality, memory capacity or useful throughput. No GPU profiling or deployment benchmark was executed for these synthetic examples.
What interviewers probe next
- "Does smaller TP mean fewer collective calls?" Not necessarily. It changes group size and per-rank work; the layer may still issue the same number of calls. Measure the resulting critical path.
- "Why not enable speculation everywhere?" Drafting, verification, memory and rejected work can cost more than the accepted-token gain, especially under different load or acceptance conditions.
- "Can a graph eliminate a 40% gap?" Only the portion caused by work it replaces. Waiting for a dependency, another rank or the client requires a different remedy.
- "Is a tenth of the bound normal?" There is no universal fraction. Verify the bound's scope and compare the actual implementation and workload before making that judgment.
Common mistakes
- Treating active parameter bytes as a measured HBM-traffic counter.
- Calling separate expert tensors inherently uncoalesced reads.
- Adding overlapped rank durations into a serial step budget.
- Omitting drafting and verification costs from a speculation speedup estimate.
Key takeaways
- The synthetic 5 ms memory floor versus 20 ms observed step identifies an unexplained gap, not its cause.
- Reducing a 9 ms step to 8 ms gives a 1.125× rate improvement under the stated one-token assumption.
- In the speculation example, acceptance 0.8 yields 2.34×, while 0.2 yields 0.70× with the same cycle costs.
- Optimize the demonstrated critical path and retest the full service contract.