TL;DR: First define what is twice as slow: local compute, input preparation, transfer, or a full step that includes waiting. A late rank can delay a synchronous job, but collectives are not all identical barriers and full-step timers can hide the cause. Compare matched phases, then test the leading hypothesis with the smallest useful change.
How to approach it
Confirm workload and timing comparability. Locate the late phase and rank before collecting every counter. Then correlate clocks, topology, host resources and device health with that phase, preserving evidence before a disruptive restart.
A strong answer
Consider a 128-node run whose steps have grown from 400 ms to 780 ms. Every GPU still appears healthy and busy. Those counters cannot distinguish a worker that is late doing its own computation from one that is waiting for data or another worker. Start with the sequence of work and waits described in Stragglers and Hangs, then use device counters to investigate the delay you found.
For the diagram's simplified rendezvous model:
127 nodes finish local work at 400 ms; one finishes at 780 ms
Mean local-work duration = (127 × 400 + 780)/128 = 402.96875 ms
Mean increase = 2.96875/400 ≈ 0.7422%
Job completion boundary = 780 ms
Throughput ratio = 400/780 ≈ 51.282%; loss ≈ 48.718%
Sanity: the mean local-work duration is not the job completion time.
A rank is one worker process in the distributed job. If its step timer includes waiting for peers, fast and slow ranks can both report about 780 ms. Record when input becomes ready, when computation finishes, and when communication is submitted and completed. GPU work often continues after the CPU submits it, so a timer around the submission call does not measure the work’s full duration. NCCL's collective definitions describe operations, but their critical-path effect depends on streams, dependencies and overlap; avoid treating every call as the same host barrier.
First compare shapes, tokens, batch sizes, rank roles, model partition, software versions and cache state. A rank processing a larger shard is not necessarily running slower per unit of work. Normalize the relevant phase by its workload before calling the node faulty. Check whether the delayed identity is a logical rank, physical device or a shared path.
| Observed phase | Next evidence |
|---|---|
| Local compute late | Matched kernel shapes, clocks, power/thermal reasons, device activity |
| Input ready late | Loader queue, CPU contention, storage latency, data-shard size |
| Host/device copy slow | NUMA placement, pinned memory, PCIe path and concurrency |
| Aligned collective arrivals, slow transfer | Group topology, selected NIC/path, link counters and contention |
| Device errors or recovery state | Supported diagnostics and vendor recovery guidance |
DCGM and GPU Telemetry supplies useful context. A lower clock with a throttle reason supports a constraint hypothesis, but half the clock does not imply exactly half the whole-step throughput. Memory-bound phases and overlap change the result. Similarly, slow collectives with normal compute can reflect late peers or scheduling, not only a bad cable.
Prefer a small matched test on suspect and known-good nodes when it can distinguish the hypotheses. Keep workload shape, software and placement comparable. If a full job restart is necessary, capture traces first and account for its cost. A node exclusion changing performance is evidence, but it also changes rank mapping, caches and routes; one restart does not perfectly separate hardware from software.
Check host CPU and memory pressure, process placement, leftover workloads, storage clients and version drift when the delayed phase points there. GPU health investigation should happen immediately when recovery signals require it; there is no universal rule that it must always be the fifth or cheapest check. The evidence determines the order.
For unexplained recurring slowness, quarantine or restrict the node according to service risk while preserving a reproducible case. Escalate diagnostics and compare environmental conditions. Retirement is a decision supported by persistent evidence and repair economics, not the automatic result of passing a health test while running slowly.
A delay that follows the data shard or worker role across matched devices points away from a defective card. A matched workload that repeatedly slows down on the same physical path points toward that path. When several nodes degrade together, investigate shared software, rack resources and workload changes before opening separate card-replacement tickets.
What interviewers probe next
- "Why can full-step times be equal?" Fast ranks include waiting for the late dependency.
- "Does low clock prove the root cause?" Correlate its reason and phase impact; it can be an effect or an unrelated constraint.
- "What makes a node-swap test useful?" Controlled workload and mapping, plus repeated evidence rather than one changed run.
Common mistakes
- Inferring useful work from a busy-device percentage.
- Calling every slow collective a network fault.
- Restarting the job before preserving the trace that explains the delay.
Key takeaways
- In the model, local mean time rises 0.742% while useful throughput falls 48.718%.
- Measure phase readiness and waiting, not only the full step.
- Choose the next check from evidence, not a fixed five-command ritual.