← Coding for Infra / 32

hardNew

Implement beam search without losing finished hypotheses or corrupting KV ownership

Keep live and completed candidates separate, score whole sequences, and trace a case where greedy decoding loses. Explain the cache cost of branching.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Keep live and completed candidates separate, score whole sequences, and trace a case where greedy decoding loses. Explain the cache cost of branching.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 303 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.

🚀 Inference & Serving

The KV CacheThe KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.

🧮 Napkin Math & Capacity

KV Cache SizingDerive cache payload from the stored attention state, then account for mixed sequence lengths, block allocation and per-device limits. Worked examples distinguish exact bytes from deployable capacity.

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackImplement a LoRA forward pass and merge two adapters without inventing cross terms Next in this trackImplement scalar autograd and prove that a shared input receives every gradient contribution Next in this trackImplement a bounded request log with timers that remain correct under concurrency

DISCUSSION · 0

No comments yet — be the first to share your approach.