timeouts

The watchdog says an all-reduce ran for ten minutes and never finished. That means one rank never entered it, entered a different one, or died. How to read the log line, the flight recorder that names the rank in seconds, the env vars that make the next one debuggable, and why the timeout is a symptom every time.

Four of them diagnose, four configure the machine correctly, and several popular ones override tuning that was already right. Which is which, the symptom each addresses, and why a cluster whose job scripts carry a dozen of these has a configuration problem rather than a tuning one.

A collective stops when one participant stops, and a job of a thousand ranks has no way to distinguish a link that will come back in two seconds from a rank that has died. What the job actually does during those seconds, the cost per flap, and why the fault passes every test you would run.

A hang is a collective that one rank never entered, and finding it means asking which rank is missing rather than what is broken. The flight recorder that answers that in seconds, what to do when it is not enabled, and the four causes that a missing rank turns out to be.