expert★ EssentialNewOpenAIAnthropicTogether AI

Design an inference platform for 10,000 requests per second on a 70B model. Size it and name the SLOs.

Ten thousand requests a second is a fleet of hundreds of nodes, a router that has to know what every replica is caching, and an SLO pair that decides the batch on every one of them. Here is the sizing chain, node by node.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

📐 AI Systems Design

Inference Platform ArchitectureAn LLM inference platform is the layer between a product's API call and a GPU running a serving engine, and every design round starts from its reference shape: a gateway that authenticates and rate-limits, a router that picks a replica with the right model and a warm cache, a per-replica scheduler that batches, engines that run prefill and decode, a KV cache tier, an autoscaler, and the observability that makes it operable. This page draws that shape, sizes each box for a concrete workload, and walks the derivation from user demand to replica count that every design answer has to contain.

Foundational

🧮 Open Weights & Serving Engines

Capacity Planning for Open-Weights FleetsPlanning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.

Foundational

🧭 Ownership & Judgment

Talking About Cost and Capacity with LeadershipInfrastructure engineers are asked to justify large numbers to people who do not share their vocabulary, and the conversations go wrong in predictable ways: a technical objection with no alternative, a forecast with no assumptions, or a cost quoted in a unit the listener cannot act on. What works is a small number of costed options, a stated recommendation, the decision needed by a date, and every figure expressed in whatever the listener actually controls.

Advanced

🚀 Inference & Serving🔒 Premium

Inference Autoscaling and Cold StartsScaling an LLM fleet is harder than scaling a web service because a replica takes minutes to become useful (pull an image, load 141 GB of weights, warm the cache) and costs several dollars an hour while idle. The signals that work are queue depth and TTFT against the SLO, not GPU utilization, which is misleading for memory-bound decode. The design is a warm pool sized for the burst, hysteresis so the fleet does not thrash, and a cold-start path measured in seconds through snapshots and weight streaming.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicWalk me through an inference platform for a hosted LLM. What are the pieces, and what does each one do?Closest to this topicOne fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.Closest to this topicDesign autoscaling for an LLM service whose traffic triples within a minute. What signal, what cold-start budget, what warm pool?

FEDITOR'S NOTE

Scored on the sizing chain (tokens per second, prefill FLOPs, decode bandwidth, KV capacity, then nodes with headroom), on a router and scheduler that keep prefix hits and goodput up, and on a cost number at the end with the assumptions visible.

DISCUSSION · 0

No comments yet — be the first to share your approach.