mfu

Acceptance measures the hardware and production measures the whole system, so the gap lives in everything acceptance did not cover. Four candidate layers, the order to eliminate them, and the two that account for most of these cases.

Compute the utilization first so you know how much time is unaccounted for, then read one timeline in a fixed order: GPU idle, then what the host was doing in the gaps, then whether the gradient all-reduce overlapped the backward pass, then the optimizer. Four causes, the evidence for each, and what each is worth.

The one utilization number that cannot be gamed by recompute: model FLOPs over hardware peak, derived from a step time in four lines, with the band frontier labs land in and an itemized list of where the other 60% goes.

Full recomputation costs a third more compute to cut activation memory by an order of magnitude. The per-layer arithmetic that says whether you need it at all, the selective variant that costs almost nothing, and the point at which a smaller micro-batch is the cheaper way out.

Compute the step time the formula predicts, measure the one you have, and the gap is the budget to explain. The isolation order with one metric per suspect: dataloader wait, SM and tensor-core activity, HBM throughput, time in collectives, and the per-rank spread that says it is one machine.

The fleet equation applied to 6.35e24 FLOPs: 11 days, and how the answer swings from 9 to 15 with the one parameter the interviewer wants you to state.

The fleet equation solved for GPU count: about 6,200 H100s at 40% MFU, why it rounds up to a power of two, and the reasons a good answer adds 15% before naming a number.

6ND on 15.6 trillion tokens gives 3.8e25 FLOPs; converting to GPU-hours needs an MFU, and Meta's published 30.8 million hours lets you solve for it. The chain, the check, and what the implied MFU tells you.

Dollars per FLOP from the GPU price and the MFU, times 6N: a 70B training token costs about three quarters of a microdollar, and the whole 15T-token run follows in one multiplication. The chain, the comparison to an inference token, and why the training token is cheaper.