hard★ EssentialNewOpenAIBasetenTogether AI

Size an inference fleet for a 70B model serving 1,000 concurrent users

Two constraints per replica, memory and throughput, each derived from the model and the traffic; whichever binds sets the users per replica, and the replica count follows with headroom. The full chain from 1,000 users to a number of nodes.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Core

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainThe fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Cost per Million TokensThe unit every serving decision cashes out in. It is one formula: the fleet's dollars per second divided by the tokens per second it sustains, scaled to a million, with utilization in the denominator because idle replicas still cost money. This page derives it from a GPU price and a throughput estimate, works it at three batch sizes to show why batching is the main lever, separates prefill from decode pricing, and shows how the same fleet's cost per token moves by 5x between a quiet hour and a busy one.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Capacity Planning and UtilizationCapacity planning for GPUs is deciding how many to have next quarter given that they cost money whether busy or not, that demand arrives in bursts, and that a queue near saturation produces waits that grow without bound. This page works the planning arithmetic for a serving fleet (peak demand, headroom, the p99 penalty of running hot) and a training platform (job mix, queue wait, the value of a shared pool), and gives the queueing intuition that makes 70% look full. The number that decides everything is utilization, and it has a ceiling set by latency, not by hardware.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?Next in this trackEstimate how long it takes to write a checkpoint for a 405B training run Next in this trackWhat is the power budget for a cluster of 1,024 H100s?

FEDITOR'S NOTE

A staff-level answer computes users per replica from both the KV budget and the token-rate budget, says which one binds, and then adds headroom with a reason. Skipping either constraint is the common failure.

DISCUSSION · 0

No comments yet — be the first to share your approach.