AI Infra Interviews logo
← LLM Inference & Serving / 12
mediumSGLangAnthropicOpenAI

Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?

Two requests share 2,000 leading tokens but ask different questions afterward. Follow the shared state through a branch, calculate the reusable blocks, and distinguish memory savings from measured latency savings.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Two requests share 2,000 leading tokens but ask different questions afterward. Follow the shared state through a branch, calculate the reusable blocks, and distinguish memory savings from measured latency savings.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced
🚀 Inference & Serving🔒 Premium
Prefix Caching and KV ReusePrefix caching reuses compatible attention state for leading tokens. Calculate shared memory and token reuse, then weigh cache identity, block ownership and routing delay before expecting a latency gain.
Advanced
🚀 Inference & Serving🔒 Premium
Hybrid State Checkpoints and RollbackA hybrid model must restore attention history and recurrent state to the same token boundary. Learn why a cache hit can require replay, how rejected draft tokens corrupt an in-place state, and what saved checkpoints cost.
Advanced
📐 AI Systems Design🔒 Premium
Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.
Advanced
🚀 Inference & Serving🔒 Premium
The Serving Optimization WaterfallTune an LLM deployment one change at a time against one fixed workload, in an order that follows the binding constraint. A measured run went from 789 to 6,568 output tokens per second, and the two biggest wins were not the knobs people usually reach for first.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackYou have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.Next in this trackYour p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.Next in this trackvLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?
FEDITOR'S NOTE

Trace a shared prefix and its two branches through cache lookup and physical ownership. Explain compatibility, active references and eviction without assuming that every repeated prompt produces a hit.

DISCUSSION · 0

No comments yet — be the first to share your approach.