router

AI infra interview questions tagged router, across every topic.

2 questions · 1 unlocked for you

Open full answer →

Open full answer →

Concepts behind "router"

The curriculum that explains the ideas these questions test.

📐 AI Systems Design

Inference Platform ArchitectureAn LLM inference platform is the layer between a product's API call and a GPU running a serving engine, and every design round starts from its reference shape: a gateway that authenticates and rate-limits, a router that picks a replica with the right model and a warm cache, a per-replica scheduler that batches, engines that run prefill and decode, a KV cache tier, an autoscaler, and the observability that makes it operable. This page draws that shape, sizes each box for a concrete workload, and walks the derivation from user demand to replica count that every design answer has to contain.

Explore the full curriculum →