Advanced

Speculative Decoding

Decode is memory-bound: each step reads every weight to produce one token. Speculative decoding has a cheap draft propose several tokens, then verifies them all in one forward pass of the big model, so one weight read yields several tokens with output distribution unchanged. It wins 2x to 3x at small batch, breaks even near the ridge point where the GPU is already compute-bound, and lives or dies on the acceptance rate, which is what interviewers ask you to reason about.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Prefill vs Decode Memory-Bound vs Compute-Bound Kernels Bandwidth-Bound Decode Throughput The KV Cache Serving Engines: vLLM, SGLang and TensorRT-LLM Roofline Model

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingEAGLE, Medusa or a separate draft model: which speculative decoding method do you pick, and why?→LLM Inference & ServingWhen does speculative decoding speed up serving, and when does it break even or hurt?→AI Infrastructure System DesignDesign speculative decoding into a production serving fleet: draft placement, acceptance monitoring, and the batch regime where it pays.→Open-Weights Models & Serving EnginesIs speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?→Coding for InfraSimulate speculative decoding to find the expected tokens per round and the batch size where it stops paying.→Napkin Math, Cost & CapacityHow many tokens per second can a 70B model generate for a single user on H100s?→