TL;DR: Preserve the evidence before restarting. Compare the last useful progress and collective activity across ranks, then investigate the first divergence. A missing participant is one possible cause; all ranks can also enter the same collective and stall during execution. Use traces, host stacks and device or network events together.
How to approach it
Establish whether the job has stopped advancing or is taking an unusually long time in a legitimate phase. Locate the affected process group and operation, compare rank activity, and trace the earliest unexplained delay. Keep the diagnosis separate from the bounded recovery action needed to restore the job.
A strong answer
In distributed training, a collective is a communication step every rank in a process group (the set of ranks that communicate together) must join, such as an all-reduce, so one missing rank stalls the rest. Suppose a run stops advancing every few hours. Restarting restores progress but leaves no explanation. Before the next restart, capture the job attempt, last completed step, process membership, collective traces and relevant host logs. GPU activity alone cannot locate the fault: a waiting communication kernel may appear busy, while a host-side wait may leave the device idle.
Stragglers and Hangs explains the distinction between delayed progress and a complete stall. Start with a timeline: when did the application last finish useful work, and which dependency has not completed since then?
Compare operation records within the right group. PyTorch's Flight Recorder tutorial describes tracing distributed operations for hang diagnosis. Configure and test collection before a long run. The ProcessGroupNCCL settings include a bounded trace buffer and options for timeout dumps, timing and stack collection. Verify the settings for the deployed framework version; enabling a dump flag without the required trace buffer is not a complete setup.
The recorder can show which collective was enqueued, started or completed on a rank, depending on configuration. Compare process-group identity, membership, sequence, operation and expected shapes. Sequence numbers from unrelated process groups cannot simply be lined up as one global clock. A missing record may mean a process failed or its dump was unavailable, so retain that uncertainty.
| Observation | Next investigation |
|---|---|
| One rank has not issued the expected operation | Its preceding compute, input wait, lock or process failure |
| Ranks issue incompatible operations | Divergent control flow, inconsistent shapes or group membership |
| All issue a matching operation, but it does not finish | GPU dependencies, transport progress, device health and execution errors |
| Last operations completed everywhere | Host-side waits, the next input or checkpoint phase, and other process groups |
An enqueued collective may still wait for an earlier GPU operation. Likewise, matching host-side records do not prove that bytes moved successfully. NCCL and Collective Algorithms explains the participating-rank contract; it should not be interpreted as saying every collective is a host-side barrier or that every hang has an absent rank.
Consider this illustrative trace for one process group:
rank 0: operation 420 completed; operation 421 not recorded
rank 1: operation 420 completed; operation 421 enqueued, incomplete
rank 2: operation 420 completed; operation 421 enqueued, incomplete
rank 3: operation 420 completed; operation 421 enqueued, incomplete
Rank 0 is a useful place to start, provided its trace is complete. Inspect what it does between operations 420 and 421. A stack in a data-loader read suggests an input dependency; a vanished process calls for exit status and host logs. Neither observation alone proves the ultimate cause. If rank 0 did enqueue 421 and all four records are incomplete, continue into device dependencies and transport instead of declaring the fabric healthy.
Use stacks to explain the trace. Capture Python and native stacks through tooling already supported by the job. An arbitrary signal is unsafe unless a handler has been registered for it. Sample several waiting ranks as well as the suspected outlier. A host stack inside a communication call describes where that thread is waiting; it may not identify which earlier GPU operation prevented progress.
A rank-conditional branch is a useful failure to test. Suppose rank 0 computes a metric that internally calls all-reduce, while the other ranks proceed directly to backward. The metric collective has no matching call on its peers. Compute the distributed metric on all required ranks, then restrict only the printing to rank 0. This defect can reproduce at two ranks when the branch is exercised; scale is not required for it to exist.
Collect evidence from the other layers. Check process exits, memory exhaustion, driver events, NIC/link counters, input-worker state and storage requests around the first divergence. A network error makes a transport hypothesis stronger, but a quiet counter does not rule it out. A single slow storage request can delay one rank and make the resulting wait appear to be a communication problem.
Recovery must have a limit. Preserve available dumps, stop the affected attempt using the framework's supported path, reconcile its resources and resume from a verified checkpoint. Do not keep restarting a deterministic application bug indefinitely. A dump destination that shares the failed storage dependency may be unavailable, so test local capture and later collection as part of the runbook.
Choose timeout and progress-alert thresholds from measured legitimate phase durations. Long checkpoints and unusually large collectives need different allowances from ordinary steps. A host heartbeat proves that its writer ran; track completed training progress as well. Measure tracing overhead, buffer coverage and dump reliability under a controlled failure. There is no universal promise of zero overhead or a diagnosis in ten minutes.
What interviewers probe next
- “All ranks have the same sequence number. Is communication cleared?” No. Check completion, expected arguments, GPU dependencies and transport progress within that group.
- “The recorder was not enabled?” Preserve logs and supported stack dumps now, then reproduce with bounded tracing. Some evidence from the failed attempt may be unavailable.
- “Should we increase the timeout?” Only if legitimate operations need it. A longer timeout can hide a recurring fault while increasing wasted capacity.
Common mistakes
- Treating every hang as a missing participant.
- Comparing sequence numbers without process-group identity or trace completeness.
- Restarting before collecting evidence when collection is possible.
- Using a host heartbeat as proof that training advances.
Key takeaways
- Compare enqueue, execution and completion rather than a sequence number alone.
- In the four-rank example, the missing record narrows the investigation; it does not establish a cause.
- Test capture and recovery before the long run, including failure of the dump destination.