data pipeline

A loss that stops moving has five different causes and one of them is not a training problem at all. The checks in order: is the step counter moving, what are the learning rate and the grad norm doing, is the data repeating, has the loss scaler collapsed, and is one rank sending zeros.

A 400B dense model on 10 trillion tokens is 2.4 × 10²⁵ FLOPs, 43 days of pure compute on 16,384 H100s, and about 380 interruptions along the way. The order in which to derive every number, the layout and the data rate, the checkpoint and failure budgets, and the schedule that survives its own arithmetic.

Fifteen trillion tokens starts as a few petabytes of raw text and ends as 30 TB of shards a training job reads at 1.7 million tokens per second. The stages, the bytes at each boundary, the throughput each one needs to finish in two weeks, and the two stages where the pipeline actually spends its time.