← AI Infrastructure System Design / 29

mediumDatabricksSnowflake

Design an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.

Define model aliases, permission-aware routing, response reuse and bounded retries. Trace a request whose cached answer becomes invalid after its source permissions change.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Define model aliases, permission-aware routing, response reuse and bounded retries. Trace a request whose cached answer becomes invalid after its source permissions change.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.

📐 AI Systems Design

Inference Platform ArchitectureAn inference platform connects API requests to model replicas. Learn how routing, batching, memory limits and scaling work together to meet a latency target.

📐 AI Systems Design🔒 Premium

Model Routing and CascadesChoose a model against measured task quality, latency and cost. Distinguish routing before generation from cascades that pay for a first answer before escalating.

🩺 Fleet Reliability & Observability🔒 Premium

Tracing Inference RequestsConnect gateway, queue, model execution and streaming delivery without confusing overlapping spans or GPU activity with user-visible latency.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign inference for a 1T-parameter mixture-of-experts model: expert parallelism on an NVL72 rack, the attention cache, balance.Next in this trackDesign a training cluster for a one-trillion-parameter MoE. Size it, choose the parallel layout, and map it onto the fabric.Next in this trackDesign an asynchronous batch inference API for a billion records with a completion deadline

FEDITOR'S NOTE

Explain which guarantees the gateway enforces before choosing a provider. A cache key, fallback and degraded mode must preserve those guarantees.

DISCUSSION · 0

No comments yet — be the first to share your approach.