Advanced

Request Routing and Load Balancing for LLMs

An LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Inference Platform Architecture Prefix Caching and KV Reuse Continuous Batching Capacity and Backpressure Inference Autoscaling and Cold Starts Rate-Limiting Algorithms

LESSONS THAT TEACH THIS

Inference EngineeringChoose an engine against explicit responsibilities14 min→Inference EngineeringPrefix reuse saves prompt work and shared storage when state matches14 min→Inference EngineeringRoute models before replicas22 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?→Distributed Training & ParallelismYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?→AI Infrastructure System DesignDesign a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.→LLM Inference & ServingYour p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.→Open-Weights Models & Serving EnginesOne tenant's traffic is pushing everyone else past their latency objective. What do you do?→AI Infrastructure System DesignDesign an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.→