capacity planning

The H200 has the same compute die as the H100 and costs more per hour. The datasheet gives two ratios, 1.43x bandwidth and 1.76x memory, and those two numbers decide exactly which workloads pay back the premium and which ones lose money on it.

A Trn2 instance carries 16 Trainium2 chips with 1.5 TB of HBM and more dense bf16 FLOPS than an 8-GPU H100 node. Whether the cheaper FLOPS reach your workload depends on the Neuron compiler, the kernels you do not have yet, and an MFU number you must measure rather than assume.

Two constraints per replica, memory and throughput, each derived from the model and the traffic; whichever binds sets the users per replica, and the replica count follows with headroom. The full chain from 1,000 users to a number of nodes.

Peak sets the fleet, average sets the bill, and the ratio between them is idle money. The chain from a 3x diurnal peak to a replica count, the utilization it implies, what autoscaling can and cannot recover given model load times, and what to do with the trough.

Port counts decide everything: a two-tier Clos of 64-port switches tops out at exactly 4,096 endpoints at full bisection, which is why that number appears in so many cluster designs. The switch and optics count both ways, what oversubscription actually slows down, and where the money goes.

The number comes from one requirement: the gradient reduction has to finish inside the backward pass that produces it. Working that requirement backward gives a bandwidth per GPU, and the answer lands near the port speed the industry ships, which is not a coincidence.

Committed capacity is cheaper per hour and cannot shrink, so the decision is a utilization break-even, not a discount. The arithmetic for where the reserved baseline should sit, why the p50 of steady demand is the target, and how on-demand, spot and capacity blocks absorb everything above it.

Three quantities decide it: how often the job is interrupted, how long a checkpoint takes, and how long a restart takes. The formula that turns them into an optimal interval, the goodput it leaves, and the measured result showing that making checkpoints faster is worth more than checkpointing more often.

Three separate reserves get merged into one number and then argued about. The repair pipeline from Little's law, the restart pool that has to be instantly available, and the correlated-failure buffer sized by the largest thing that can fail at once, each derived and then added.