mediumNewTogether AI

Simulate speculative decoding to find the expected tokens per round and the batch size where it stops paying.

A geometric process with a cap, a closed form that a twenty-line simulation confirms, and a speedup that turns negative when the verification stops being free. The table across acceptance rates and draft lengths, and the crossover from a 2.35 times win to a 0.75 times loss under two stated cost regimes.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

Speculative DecodingDecode is memory-bound: each step reads every weight to produce one token. Speculative decoding has a cheap draft propose several tokens, then verifies them all in one forward pass of the big model, so one weight read yields several tokens with output distribution unchanged. It wins 2x to 3x at small batch, breaks even near the ridge point where the GPU is already compute-bound, and lives or dies on the acceptance rate, which is what interviewers ask you to reason about.

Advanced

💻 Coding for Infra🔒 Premium

Batching Queues and BackpressureWrite a request batcher is the coding round's version of the serving engine's scheduler: requests arrive one at a time, the GPU wants them in groups, and the batcher decides when a group is full enough to send without holding anyone too long or accepting more than it can hold. The two knobs are the maximum batch size and the maximum wait, the invariant is a bounded queue, and the follow-ups (priorities, cost-aware batching, cancellation, bounded in-flight batches) are the ideas the real engines carry. This page implements the batcher in asyncio, derives what each knob buys, and walks the follow-ups.

Core

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Advanced

🚀 Inference & Serving🔒 Premium

Multi-LoRA ServingA LoRA adapter is a few hundred megabytes of low-rank matrices that turn a base model into a fine-tuned variant, and multi-LoRA serving runs hundreds of them on one copy of the base weights by keeping the adapters in memory and applying the right one per request inside the batch. It is how a platform serves a thousand customers' fine-tunes without a thousand deployments. The costs are an extra small matmul per layer, adapter memory and loading, and a scheduler that has to batch across adapters without starving any of them.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the API for a GPU job scheduler in 45 minutes. What are the resources, states and semantics?Next in this trackImplement dominant resource fairness for a cluster where jobs need different mixes of GPU, CPU and memory.Next in this trackWrite a tiled matrix multiply and explain the cache effect. Then say honestly what your benchmark actually measured.

FEDITOR'S NOTE

Scored on the closed form matching simulation, on the speedup expression including draft and verification cost, and on the sign change between small and large batch, including whether the candidate says which inputs are assumed and which are measured.

DISCUSSION · 0

No comments yet — be the first to share your approach.