timeouts

Read a watchdog timeout without blaming the reporting rank. Compare group-scoped traces, separate missing evidence from mismatched calls, and distinguish CPU submission from GPU completion.

Separate NCCL logging, device selection and execution overrides from PyTorch failure handling. Work through a faster collective that makes the job slower, and learn which timeout actually fired.

Trace a link flap through transport and job recovery. Calculate detection, restart and replay costs, and learn what continuous counters and timed tests can actually establish.

Find where useful progress stopped by comparing per-rank traces, host stacks and device events. Distinguish late arrival, mismatched collectives and stalled execution before choosing a recovery action.