training

The roofline, the sharding arithmetic and the parallelism trade-offs transfer unchanged, and the numbers are in the same units. What changes is who writes the kernels, how shapes must behave, and the interconnect topology you shard against. The v6e and v7 numbers worked through against an H100.

Capacity is the easy half and almost never the constraint. The number that sizes the system is the checkpoint write burst, which is thousands of times the steady dataset read, and the tier that serves each is different. The full derivation, and the tiering that avoids paying burst prices for archive bytes.

Compute the utilization first so you know how much time is unaccounted for, then read one timeline in a fixed order: GPU idle, then what the host was doing in the gaps, then whether the gradient all-reduce overlapped the backward pass, then the optimizer. Four causes, the evidence for each, and what each is worth.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

The fleet equation applied to 6.35e24 FLOPs: 11 days, and how the answer swings from 9 to 15 with the one parameter the interviewer wants you to state.

The fleet equation solved for GPU count: about 6,200 H100s at 40% MFU, why it rounds up to a power of two, and the reasons a good answer adds 15% before naming a number.

Sixteen bytes per parameter is 1.1 TB of static state before a single activation; LoRA keeps the 141 GB of frozen weights and shrinks the rest to a few gigabytes. The derivation, the GPU counts, and the activations that both still pay.

Gradient bytes per step over step time, with the ring all-reduce factor that nearly doubles it: the arithmetic for a 7B and a 70B, the link speeds that cover it, and the step-time threshold where overlap stops hiding it.

6ND on 15.6 trillion tokens gives 3.8e25 FLOPs; converting to GPU-hours needs an MFU, and Meta's published 30.8 million hours lets you solve for it. The chain, the check, and what the implied MFU tells you.

Memory follows total parameters and compute follows active ones, so the two models pull the GPU count in opposite directions. The chain for a 235B-total, 22B-active MoE against a dense 70B, for serving and then for training.

Sixteen bytes per parameter is 6.5 TB per checkpoint; over a 100 GB/s filesystem that is about a minute, which at a 30-minute cadence is 3% of the run. The chain, the storage tiers that meet it, and why the write is made asynchronous.

The ring all-reduce formula, 2(n−1)/n times the buffer over the link bandwidth, applied to 141 GB of bf16 gradients on an NVLink node: 0.27 s by the formula, and the two reasons the measured number is different.

The token stream is tens of megabytes a second and the checkpoints are tens of gigabytes a second; the two workloads differ by a thousand and want different tiers. The chain from tokens per second to bytes, the checkpoint burst, and the restore that sets the real spec.

Dollars per FLOP from the GPU price and the MFU, times 6N: a 70B training token costs about three quarters of a microdollar, and the whole 15T-token run follows in one multiplication. The chain, the comparison to an inference token, and why the training token is cheaper.

6ND to a compute budget, the fleet equation to a GPU count, the rental rate to a bill: 64 H100s, about 19 days, roughly $75k plus a buffer. The chain, the reasons the answer is 64 and not 59, and the memory check that says the fleet is compute-sized, not memory-sized.

Activations are the term the 16-bytes-per-parameter rule leaves out, and at 8k context they are bigger than the weights: about 34 × sequence × hidden bytes per layer, 2.3 GB, 182 GB per micro-batch across 80 layers. The chain, what FlashAttention removes, and what checkpointing does to the number and the compute.

A shared pool recovers the GPUs inference holds for its peaks, but a preempted training gang takes minutes to give them back and an SLO breaks in seconds. The utilization of each design, the reclaim-time arithmetic against the traffic ramp, and the floor-plus-borrow split most fleets land on.

A training run has one number that matters and a handful that explain it. Goodput as the top-level metric, the per-rank timing that finds a straggler among a thousand, the hardware layer beneath it, and the three dashboards that serve three different people asking three different questions.

A three-month run on 8,000 GPUs is a large bet against everything that can happen to one building. The recovery point and the recovery time as numbers, the checkpoint replication that sets the first, the second cluster and its warm state that set the second, and the drills that make the numbers true.