Advanced

Hybrid State Checkpoints and Rollback

A hybrid model must restore attention history and recurrent state to the same token boundary. Learn why a cache hit can require replay, how rejected draft tokens corrupt an in-place state, and what saved checkpoints cost.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Prefix Caching and KV Reuse Speculative Decoding KV Cache Sizing Asynchronous GPU Buffer Ownership Reading config.json to Size a Model You Have Never Run

LESSONS THAT TEACH THIS

Inference EngineeringAttention architecture determines which state the serving engine must retain29 min→

PRACTICE THIS IN REAL QUESTIONS

CUDA, Triton & Kernel EngineeringImplement the verification step of speculative decoding, including the rollback of the KV cache after a rejection.→Open-Weights Models & Serving EnginesServe Kimi K3, a 2.8 trillion parameter model. What is the minimum viable configuration?→LLM Inference & ServingWhen does speculative decoding speed up serving, and when does it break even or hurt?→Open-Weights Models & Serving EnginesA model interleaves linear-attention and full-attention layers. What changes about serving it?→LLM Inference & ServingHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?→Open-Weights Models & Serving EnginesIs speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?→