Build a reference-counted KV block allocator and test that forked sequences keep their data independent when either branch appends.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Coding for Infra / 14
hardvLLMTogether AI
Implement the block allocator behind a paged KV cache, including prefix sharing and copy-on-write.
Build a reference-counted KV block allocator and test that forked sequences keep their data independent when either branch appends.
Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Advanced
Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.📐 AI Systems Design🔒 Premium
Foundational
The KV CacheThe KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.🚀 Inference & Serving
Foundational
KV Cache SizingDerive cache payload from the stored attention state, then account for mixed sequence lengths, block allocation and per-device limits. Worked examples distinguish exact bytes from deployable capacity.🧮 Napkin Math & Capacity
Advanced
Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.🧮 Napkin Math & Capacity🔒 Premium
UP NEXT ON YOUR JOURNEY
Next in this trackImplement the scheduler loop of a continuous-batching engine: admit, step, preempt. Where does it livelock?Next in this trackIn Go, download many model shards concurrently with a concurrency limit, a timeout and clean cancellation. Write it.Next in this trackRoute requests to replicas by prefix using a consistent hash ring. Why virtual nodes, and how many?
FEDITOR'S NOTE
Look for ownership checks on every append, executable payload isolation tests, correct tail-waste arithmetic and safe release lifetimes.
DISCUSSION · 0
No comments yet — be the first to share your approach.