hangs

A loss that stops moving has five different causes and one of them is not a training problem at all. The checks in order: is the step counter moving, what are the learning rate and the grad norm doing, is the data repeating, has the loss scaler collapsed, and is one rank sending zeros.

The watchdog says an all-reduce ran for ten minutes and never finished. That means one rank never entered it, entered a different one, or died. How to read the log line, the flight recorder that names the rank in seconds, the env vars that make the next one debuggable, and why the timeout is a symptom every time.

A hang is a collective that one rank never entered, and finding it means asking which rank is missing rather than what is broken. The flight recorder that answers that in seconds, what to do when it is not enabled, and the four causes that a missing rank turns out to be.