Advanced

Speculative Decoding

A draft proposes tokens; the target checks them with a sampling rule that preserves its distribution. Work through the correction after rejection, count accepted drafts separately from emitted tokens, and decide whether the measured round cost earns a speedup.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Prefill vs Decode Memory-Bound vs Compute-Bound Kernels Bandwidth-Bound Decode Throughput The KV Cache Serving Engines: vLLM, SGLang and TensorRT-LLM Roofline Model

LESSONS THAT TEACH THIS

Inference EngineeringSpeculation must repay the whole round19 min→Inference EngineeringA serving benchmark needs a workload and an arrival model17 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingEAGLE, Medusa or a separate draft model: which speculative decoding method do you pick, and why?→LLM Inference & ServingWhen does speculative decoding speed up serving, and when does it break even or hurt?→AI Infrastructure System DesignDesign speculative decoding into a production serving fleet: draft placement, acceptance monitoring, and the batch regime where it pays.→Open-Weights Models & Serving EnginesIs speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?→Coding for InfraSimulate speculative decoding to find the expected tokens per round and the batch size where it stops paying.→LLM Inference & ServingWhen does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?→