flight recorder

The watchdog says an all-reduce ran for ten minutes and never finished. That means one rank never entered it, entered a different one, or died. How to read the log line, the flight recorder that names the rank in seconds, the env vars that make the next one debuggable, and why the timeout is a symptom every time.

A hang is a collective that one rank never entered, and finding it means asking which rank is missing rather than what is broken. The flight recorder that answers that in seconds, what to do when it is not enabled, and the four causes that a missing rank turns out to be.

Each rank keeps a ring buffer of its recent collectives, and on a timeout every rank dumps it. Comparing sequence numbers across a thousand of those buffers finds the missing participant in one pass, and the three patterns those comparisons produce point at three different causes.

Full GPU utilization during a hang is the clue, because a spinning collective looks identical to real work. The isolation order that finds the mismatched rank in minutes, a runnable reproducer that hangs on demand, and the fix that gates the logging rather than the collective.