hardSGLangAnthropicPerplexity

How do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?

A cached replica can still be the slower choice. Compare saved prefill against waiting time, then handle stale cache reports, pending dispatches and tenant boundaries.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

A cached replica can still be the slower choice. Compare saved prefill against waiting time, then handle stale cache reports, pending dispatches and tenant boundaries.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingConsistent hashing limits key movement when a shard set changes. It can preserve cache affinity, but it does not guarantee that cached data is present or that load is balanced.

Advanced

🚀 Inference & Serving🔒 Premium

Hybrid State Checkpoints and RollbackA hybrid model must restore attention history and recurrent state to the same token boundary. Learn why a cache hit can require replay, how rejected draft tokens corrupt an in-place state, and what saved checkpoints cost.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEExpert parallelism places different MoE experts on different devices. Routing tokens between them adds communication and makes uneven expert demand a performance concern.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou stream tokens to clients over HTTP. What can go wrong, and how does the server handle each case?Next in this trackYou need to serve 128k-token contexts. What breaks first, and what do you change?Next in this trackEAGLE, Medusa or a separate draft model: which speculative decoding method do you pick, and why?

FEDITOR'S NOTE

Compare a busy replica with a cached prefix against a quieter one without it. Work out when the saved prefill time outweighs the queue delay, then account for stale cache information.

DISCUSSION · 0

No comments yet — be the first to share your approach.