A training run hangs every few hours with no error, and the GPUs sit idle until the timeout fires. Find the cause.

TL;DR: A hang is almost always a collective mismatch: 1,023 ranks entered an all-reduce and one did not, so every rank blocks at the barrier and the job sits until the collective timeout. Reframe the question from "what is broken" to "which rank is missing and what is it doing instead", because that single change makes it tractable. The flight recorder answers it directly: it keeps a ring buffer of recent collectives per rank, and on timeout the dump shows which sequence number each rank reached, so the rank that stopped one earlier than everyone else is named. If it is not enabled, fall back to a per-rank stack dump, which shows the same thing more slowly. Then the missing rank has four possible explanations: it crashed, it is stuck in something else such as a data-loader read, it entered a different collective because the code took a different branch, or its network path failed. Each has a distinguishing check, and the fix differs completely between them, which is why identifying the rank first matters more than any hypothesis about causes.

How to approach it

Reframe first, because "the job hung" is not actionable and "rank 847 never entered collective 12,043" is. Name the instrument that gives you that in seconds and say what to do when it is missing. Then enumerate the four things a missing rank turns out to be, with the check that distinguishes each. Close with what to enable in advance, since the difference between a ten-minute diagnosis and a two-day one is entirely preparation.

A strong answer

A typical situation: a run hangs every three to five hours. The GPUs show 100% utilization because a spinning collective looks busy, no error appears anywhere, and the job eventually dies on the timeout thirty minutes later. The team restarts it and loses the evidence each time, so after a week they have five restarts and no information.

The reframe:

what a hang is:   a collective is a barrier. Every rank must call it, with matching arguments,
                  in the same order, for any rank to proceed
so a hang means:  at least one rank did not arrive at the collective the others are waiting in
what to ask:      not "why is the job stuck" but "which rank is behind, and at what"
why this matters: the second question has a mechanical answer that takes seconds to obtain,
                  and the first invites hypotheses that take days to test

Stragglers and Hangs covers the general class; NCCL and Collective Algorithms covers why the barrier semantics are what they are.

The primary instrument:

the flight recorder
  a ring buffer per rank recording recent collective operations: a sequence number, the
  operation type, sizes, and whether it started and completed
  enabled in advance through the framework's trace-buffer setting, with a dump path
  on a collective timeout, every rank dumps its buffer

reading the dump
  line up the sequence numbers across ranks
  the healthy case:  every rank's last entry has the same sequence number and is incomplete,
                     which just means they are all waiting for one that has not arrived
  the diagnostic:    one rank's last entry is a lower sequence number than everyone else's,
                     or its last entry is a different operation type
  that rank is the one to investigate, and the sequence number says which collective

what it costs to have on: a few megabytes of memory per rank and no measurable runtime cost,
  which is why it should be on by default in every long run rather than enabled after the
  first hang

The fallback when it was not enabled:

per-rank stack dumps
  send a signal that makes each process dump its Python and native stacks, or attach a
  sampling profiler to a sample of ranks
  what you are looking for: the ranks blocked inside a collective, which is most of them, and
  the one blocked somewhere else, which is the answer
  practical problem: doing this across 1,024 processes needs tooling in place beforehand, and
  a job that dies on timeout in thirty minutes gives a limited window
  so this works, and it is slower and more fragile than the flight recorder, which is the
  argument for enabling the recorder in advance

The four causes and their distinguishing checks:

The third cause deserves emphasis because it is a code bug rather than a hardware fault and it produces exactly the same symptom:

the pattern that causes it
  if rank == 0:
      metrics = compute_something(tensor)      # this calls a collective internally
      log(metrics)
  loss.backward()                              # every rank calls collectives here

  rank 0 enters an extra collective the others never call, so from that point on every rank's
  sequence number is offset by one relative to rank 0, and the next collective mismatches
the tell: the flight recorder shows rank 0 at sequence n+1 while the rest are at n, with
  different operation types, rather than one rank simply behind
the fix: every rank calls the same collectives in the same order. Gate the logging, not the
  collective: compute on all ranks, print on one
sanity: this bug is invisible in testing at two ranks if the branch happens not to be taken,
  and it is one of the most common causes of intermittent hangs in real training code

What to enable in advance, which is the actual takeaway:

flight recorder on, with a dump path on shared storage
asynchronous error handling on, so the job aborts cleanly rather than hanging forever
a collective timeout that is a few times the longest legitimate collective, not the default
a per-rank heartbeat written to a shared location, so a hang is detected in seconds rather
  than at the timeout
stack-dump tooling that can be triggered across all ranks with one command
sanity: with these five things a hang is diagnosed in ten minutes. Without them it takes days
        and usually ends with a restart and no explanation, which is the state the scenario
        above describes

The reversal condition: not every hang is a collective mismatch. A job stuck with the GPUs actually idle rather than spinning, with no collective outstanding, is a different problem: a deadlock in the training loop's own code, a filesystem that has stopped responding on every node, or a scheduler that has not delivered a resource. The flight recorder distinguishes these in the same look, since it will show every rank having completed the same last collective and none having started another. That is a useful negative result and it redirects the investigation to the host side rather than the fabric.

What interviewers probe next

• "Why do the GPUs show high utilization during a hang?" A spinning collective keeps a kernel resident, and utilization measures kernel residency rather than useful work, so it reads busy.
• "How would you detect the hang faster than the timeout?" A per-rank heartbeat with a threshold of a few step times, which turns a thirty-minute timeout into a thirty-second detection.
• "What if all ranks show the same last sequence number?" Then nobody is behind on collectives and the problem is elsewhere: a host-side deadlock or a resource that has stopped responding on all nodes at once.
• "How do you catch the code-divergence bug before production?" Run the training loop at a small rank count with a collective-consistency check enabled, which validates that every rank issues the same sequence.

Common mistakes

• Investigating "why is the job stuck" instead of "which rank is missing", which turns a mechanical lookup into an open-ended search.
• Restarting on each hang, which destroys the evidence and guarantees a repeat.
• Enabling the flight recorder after the first hang rather than by default, given it costs a few megabytes and no runtime.
• Assuming hardware, when a rank-conditional branch that calls a collective is one of the most common causes.

Key takeaways

• A hang is a collective one rank never entered; ask which rank is behind rather than what is broken.
• Detection cost: a per-rank heartbeat turns a 30-minute timeout into a 30-second detection, and the flight recorder costs a few megabytes per rank with no measurable runtime.
• The flight recorder names it from sequence numbers in seconds, and costs a few megabytes per rank to have on.
• Four causes: the rank crashed, it is blocked outside the collective, it entered a different one through a rank-conditional branch, or its network path failed.
• Enable in advance: flight recorder, asynchronous error handling, a considered timeout, per-rank heartbeats, and cross-rank stack dumps.