expert★ EssentialOpenAIAnthropicTogether AI

Design an inference platform for 10,000 requests per second on a 70B model. Size it and name the SLOs.

Translate request demand into prompt work, generated tokens and retained state, then size a measured serving configuration and its failure reserve.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Translate request demand into prompt work, generated tokens and retained state, then size a measured serving configuration and its failure reserve.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

📐 AI Systems Design

Inference Platform ArchitectureAn inference platform connects API requests to model replicas. Learn how routing, batching, memory limits and scaling work together to meet a latency target.

Advanced

📐 AI Systems Design🔒 Premium

Offline Batch Inference vs Online ServingOnline serving promises each request a latency; a batch job promises a deadline for a whole set and a checked result for every item. That changes what to optimize, when an engine's startup cost dominates, and what delivery means.

Foundational

🧮 Open Weights & Serving Engines

Capacity Planning for Open-Weights FleetsMatch forecast traffic to a measured replica operating envelope, then check failure domains, rollout capacity, readiness time and cost. Output tokens alone omit prompt work and live state; total parameter count alone does not determine a replica's throughput.

Foundational

🧭 Ownership & Judgment

Talking About Cost and Capacity with LeadershipA capacity discussion should connect technical measurements to cost, delivery dates and risk. Present the assumptions, feasible options and a recommendation, and distinguish committed capacity from actual consumption under the relevant contract.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour speech model runs faster than real time. Why do users still wait or hear gaps?Next in this trackWould request hedging improve an LLM endpoint whose p99 TTFT is too high?Next in this trackDesign moderation in a streaming inference path without checking text after users have already seen it

FEDITOR'S NOTE

Keep benchmark inputs distinct from roofline estimates. Check cache-hit scope, phase capacity, transfer bandwidth and qualified completions.

DISCUSSION · 0

No comments yet — be the first to share your approach.