Implement a stable hash ring, test which keys move, and distinguish balanced ownership from balanced serving traffic.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Coding for Infra / 17
hardSGLangAnthropic
Route requests to replicas by prefix using a consistent hash ring. Why virtual nodes, and how many?
Implement a stable hash ring, test which keys move, and distinguish balanced ownership from balanced serving traffic.
Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Foundational
Consistent Hashing and ShardingConsistent hashing limits key movement when a shard set changes. It can preserve cache affinity, but it does not guarantee that cached data is present or that load is balanced.💻 Coding for Infra
Advanced
Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.📐 AI Systems Design🔒 Premium
Advanced
Model Routing and CascadesChoose a model against measured task quality, latency and cost. Distinguish routing before generation from cascades that pay for a first answer before escalating.📐 AI Systems Design🔒 Premium
Foundational
Multi-Region Serving and FailoverMulti-region serving can improve geographic reach and survive a regional outage. It requires enough remaining capacity and a clear plan for state, routing and degraded service.📐 AI Systems Design
UP NEXT ON YOUR JOURNEY
Next in this trackImplement sampling from a model's logits with temperature, top-k and top-p. What are the numerical traps?Next in this trackCompute per-tenant GPU-hours for a billing period from a log of allocation start and stop events.Next in this trackImplement a circuit breaker in front of a model endpoint. What are the states, and what does the half-open state buy?
FEDITOR'S NOTE
Look for deterministic membership, collision handling, reproducible distribution checks and realistic prefix-cache and load tradeoffs.
DISCUSSION · 0
No comments yet — be the first to share your approach.