You serve a small, a medium and a large open-weights model. How do you route requests between them?

Separate pre-routing from a sequential cascade, charge every attempted generation, and measure checker mistakes. A lower escalation rate is not proof of higher quality, and the largest model is not a correctness oracle.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

📐 AI Systems Design🔒 Premium

Model Routing and CascadesChoose a model against measured task quality, latency and cost. Distinguish routing before generation from cascades that pay for a first answer before escalating.

Foundational

🧮 Open Weights & Serving Engines

Capacity Planning for Open-Weights FleetsMatch forecast traffic to a measured replica operating envelope, then check failure domains, rollout capacity, readiness time and cost. Output tokens alone omit prompt work and live state; total parameter count alone does not determine a replica's throughput.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverMulti-region serving can improve geographic reach and survive a regional outage. It requires enough remaining capacity and a clear plan for state, routing and degraded service.

Foundational

🗂️ Scheduling & Orchestration

Slurm vs KubernetesSlurm and Kubernetes provide different foundations for running GPU workloads. Choose the operating model and supporting components that fit the users and jobs.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou need a released bf16 model at half the footprint. Walk through quantizing it yourself.Next in this trackDesign a deployment that serves a trillion-parameter model at a million tokens of context with usable latency.Next in this trackThe open-weights model you are deploying is multimodal. What changes about serving it?

FEDITOR'S NOTE

Scored on capability filtering, full-path cost and latency, false acceptance, conditional fallback quality and production evaluation of the routing policy.

DISCUSSION · 0

No comments yet — be the first to share your approach.