hardNewOpenAIAnthropic

Traffic peaks at three times the daily average. Capacity-plan the serving fleet.

Peak sets the fleet, average sets the bill, and the ratio between them is idle money. The chain from a 3x diurnal peak to a replica count, the utilization it implies, what autoscaling can and cannot recover given model load times, and what to do with the trough.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Capacity Planning and UtilizationCapacity planning for GPUs is deciding how many to have next quarter given that they cost money whether busy or not, that demand arrives in bursts, and that a queue near saturation produces waits that grow without bound. This page works the planning arithmetic for a serving fleet (peak demand, headroom, the p99 penalty of running hot) and a training platform (job mix, queue wait, the value of a shared pool), and gives the queueing intuition that makes 70% look full. The number that decides everything is utilization, and it has a ceiling set by latency, not by hardware.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Cost per Million TokensThe unit every serving decision cashes out in. It is one formula: the fleet's dollars per second divided by the tokens per second it sustains, scaled to a million, with utilization in the denominator because idle replicas still cost money. This page derives it from a GPU price and a throughput estimate, works it at three batch sizes to show why batching is the main lever, separates prefill from decode pricing, and shows how the same fleet's cost per token moves by 5x between a quiet hour and a busy one.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

TCO: Buy vs RentWhether to buy GPUs or rent them is a utilization question dressed as a finance question. An owned H100 costs a few tens of thousands of dollars up front and a known amount per hour in power, cooling, space and operations; a rented one costs a few dollars per hour and nothing when idle. The break-even is the utilization at which the owned hourly cost, amortized over the hardware's useful life, equals the rental rate. This page builds the owned cost from parts, works the break-even, and adds the terms the simple model leaves out: depreciation risk, reserved discounts, and the price of idle capacity.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat does one training token cost?Next in this trackEstimate the latency of one decode step for a 70B model under tensor parallelism across eight H100s Next in this trackHow big can the decode batch get before the step becomes compute-bound?

FEDITOR'S NOTE

Scored on whether the candidate sizes for peak plus headroom, computes the resulting average utilization honestly, and then treats idle capacity as a cost to reduce, with the autoscale lag as the constraint. Sizing for average is the failing answer.

DISCUSSION · 0

No comments yet — be the first to share your approach.