TL;DR: Speculative decoding helps when a draft-and-verify round delivers target-distributed output faster than ordinary decoding on the same resources. Compare expected emitted tokens per round with its full elapsed cost. Acceptance, draft cost, verification shape and scheduling all matter; there is no universal batch number at which it stops helping.
How to approach it
First define the target model's sampling contract and the workload. Trace a rejected draft token so the correctness mechanism is clear. Then derive expected emitted tokens under a stated probability model, compare round cost with plain decode, and close with a matched latency and capacity experiment.
A strong answer
A small draft model proposes several tokens while the larger target checks them together. Done exactly, the accepted output follows the same probabilities as running the large model alone, so speculation changes speed, not answer quality. If the draft is often right and cheap enough, the service advances several output positions for fewer serial target passes. A proposal that is rejected still consumed work. That wasted work belongs in the cost.
Speculative Decoding checks the draft against the target model's distribution. Let q be the draft probability of a proposed token and p the target probability under the same accepted context. Exact speculative sampling accepts that proposal with probability min(1, p/q). At the first rejection, it samples a replacement from the normalized positive part of p−q and discards the later proposals. If all proposals survive, the target supplies a bonus token. This correction rule is what preserves the target distribution, as shown in the original algorithm; judging a draft token merely plausible is not enough.
The picture uses symbolic tokens and a rejection outcome, not real probabilities. The correction token changes the subsequent context, so the later draft token cannot be retained just because it looked likely under the old context. A greedy implementation has a corresponding target-argmax comparison. Distribution preservation does not mean arbitrary implementations produce bitwise-identical samples for the same random seed.
Count output progress, including the correction or bonus. Suppose k = 4 proposals and a simplified constant conditional acceptance probability a = 0.8 at each position. The probability that at least i draft tokens survive is a^i. Add the always-emitted correction/bonus position:
expected emitted tokens E = 1 + a + a² + a³ + a⁴
= 1 + 0.8 + 0.64 + 0.512 + 0.4096
= 3.3616
expected accepted draft tokens = E − 1 = 2.3616
sanity: at a = 0, E = 1; at a = 1, E = k + 1 = 5.
Calling 3.3616 the number of accepted draft tokens overcounts by one. Real acceptance can vary with position and history. Measure the survival curve and emitted-token count; a single aggregate acceptance fraction need not reproduce this geometric model.
Use hypothetical timing inputs, all measured over the same serving boundary: plain decode takes 5 ms per output position, four draft steps take 4 ms total, target verification takes 7 ms, and other exposed round overhead is 1 ms. Then:
round cost = 4 + 7 + 1 = 12 ms
speculative mean time/output ≈ 12 / 3.3616 = 3.57 ms
relative progress rate ≈ 3.3616 × 5 / 12 = 1.40×
break-even: round cost must be below E × 5 = 16.808 ms
sanity: at a = 0.5, E = 1.9375; 12 / E = 6.19 ms, worse than 5 ms.
These are arithmetic exercises, not benchmark results for a GPU. The ratio describes average progress over many rounds, not the shape of user-visible token gaps. Several accepted tokens may arrive together after a longer wait.
Batching can make verification more expensive because the target processes more positions. It can also change draft cost, cache pressure and scheduling. Bandwidth-Bound Decode Throughput provides a scoped weight-traffic model, but a weight-only roofline cannot locate a universal speculation crossover. Measure the executed kernels, attention work and competing requests. Verification is not free merely because the plain step was bandwidth-limited.
| Evidence | Decision it supports |
|---|---|
| Emitted tokens and round duration | Whether progress beats plain decode |
| Draft/target memory and temporary state | Whether the same workload still fits |
| Client TTFT and individual gaps | Whether interactive service improves |
| Qualified completions at matched arrivals | Whether the fleet benefits |
Start with one representative endpoint and a plain-decoding baseline. Hold target settings, output validity, arrivals, lengths and resource allocation fixed. Include draft weights and cache in the budget. Sweep proposal length and concurrency, retaining failures and rejected arrivals in the report. Keep speculation where its service benefit survives those tests; disable or adapt it where extra round cost outweighs progress or reduces qualified throughput.
Draft models, feature-based predictors and prompt lookup have different compute and compatibility requirements. Choose among supported implementations after measuring the task, rather than assuming one family is always more accurate or free to run. The KV Cache also needs rollback or state-management behavior for rejected candidates.
What interviewers probe next
- "Why not propose sixteen tokens?" Later positions may rarely survive, while drafting, verification and temporary state still cost resources. Measure marginal progress against marginal cost.
- "Does a high acceptance rate guarantee speed?" No. In the example, any round above 16.808 ms loses even at a = 0.8.
- "Can different tokenizers work?" Some algorithms support translation, but the simple same-token probability ratio above assumes a shared token space; use a compatible documented method.
- "What if aggregate tokens/s rises?" Check gaps, validity and qualified completions under unchanged load before accepting the change.
Common mistakes
- Counting the bonus/correction as an accepted draft token.
- Keeping proposals after a rejection changes their conditioning context.
- Inferring verification latency from a weight-only bandwidth floor.
- Quoting a universal break-even batch size or omitting draft resources.
Key takeaways
- Four proposals at constant a = 0.8 yield 3.3616 expected emitted tokens, including one correction or bonus.
- The synthetic 12 ms round beats 5 ms plain decode at a = 0.8 and loses at a = 0.5.
- Keep exact sampling correctness separate from serving-performance claims.
- Measure end-to-end progress, memory and user-visible gaps on the actual workload.