AI Infra Interviews logo
← CUDA, Triton & Kernel Engineering / 20
hardTogether AI

Implement the verification step of speculative decoding, including the rollback of the KV cache after a rejection.

Verify a drafted prefix without biasing the target distribution, then roll back rejected KV state. A six-token prefix exposes the off-by-one: nine committed tokens can have only eight valid KV entries.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Verify a drafted prefix without biasing the target distribution, then roll back rejected KV state. A six-token prefix exposes the off-by-one: nine committed tokens can have only eight valid KV entries.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced
🚀 Inference & Serving🔒 Premium
Hybrid State Checkpoints and RollbackA hybrid model must restore attention history and recurrent state to the same token boundary. Learn why a cache hit can require replay, how rejected draft tokens corrupt an in-place state, and what saved checkpoints cost.
Advanced
🚀 Inference & Serving🔒 Premium
PagedAttentionFollow a token position through a block table, watch a cache grow, and trace a shared tail through copy-on-write. Separate memory saved from throughput gained.
Advanced
🧭 Ownership & Judgment🔒 Premium
Migrations and DeprecationsA migration changes both a system and the workflows that depend on it. Explain why the move is worthwhile, how each rollout stage limits risk, what makes rollback possible, and what users need before the old path can be retired.
Advanced
🚀 Inference & Serving🔒 Premium
Speculative DecodingA draft proposes tokens; the target checks them with a sampling rule that preserves its distribution. Work through the correction after rejection, count accepted drafts separately from emitted tokens, and decide whether the measured round cost earns a speedup.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackHow do you overlap host-to-device transfers with compute, and what has to be true before the overlap actually happens?Next in this trackWhy do tensor cores accumulate in fp32 when the inputs are bf16, and why do optimizers keep fp32 master weights?Next in this trackIn an int8 or fp8 GEMM, what exactly gets scaled, where does the dequantization happen, and what breaks if you get it wrong?
FEDITOR'S NOTE

Look for actual post-processor probabilities, residual correction, explicit logit indexing and separate committed-token versus computed-KV lengths. Page ownership, stopping behavior and measured round cost complete the implementation contract.

DISCUSSION · 0

No comments yet — be the first to share your approach.