draft model

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.

All three guess tokens for the big model to check. They differ in where the guess comes from, how many candidates they verify per step, and how often the target agrees. Acceptance length per unit of draft cost is the number that decides.

Speculative decoding buys latency with spare compute, and a busy fleet has no spare compute. The speedup formula with acceptance rate and draft cost inside it, the batch size where the gain turns negative, where the draft runs and what it costs, the monitoring that catches a silent regression, and rollout per class.