TL;DR: Do not start at a node. Start by finding the scale at which the bandwidth falls, because that single fact eliminates most of the search space. Run
all_reduce_perfwithin one node, then across two, four, eight and sixty-four, and compare the bus bandwidth at each step against the rated link speed. A drop that is already present inside one node points at NVLink or at GPUDirect being off. A drop that first appears at two nodes points at the network path for one pair, which pairwiseib_write_bwthen localizes to a link or a NIC. A drop that appears only at large scale, with every pair healthy, points at congestion or at a ring that crosses the fabric badly, and the evidence for those is switch counters and the ring layout printed at startup. The four causes are distinguished by measurement rather than by inspection, and the whole sequence takes about fifteen minutes.
How to approach it
Say the localization principle first: measure at increasing scale until the number breaks, because the breaking point names the layer. Then give the commands in that order with the healthy value for each, since a check without an expected number is not a check. Then map each outcome to its cause and its fix. Close with the one cause that hides from this sequence, which is intermittency.
A strong answer
A typical situation: a job that ran fine last week now reports a third of the expected all-reduce bandwidth. The team starts reading NCCL environment variables and trying settings, which changes the number slightly in both directions and produces no diagnosis. Nothing has been measured at a scale small enough to isolate anything.
The sequence, with the number each step should return:
0. establish the target
one 400 Gb/s NIC per GPU = 50 GB/s. A healthy multi-node all-reduce reports bus bandwidth
near that per GPU. Read the bus bandwidth column, not algorithm bandwidth: bus bandwidth
already accounts for the 2(N-1)/N factor and is what compares against line rate.
1. inside one node, 8 GPUs
./all_reduce_perf -b 8 -e 8G -f 2 -g 8
healthy: bus bandwidth well above a single NIC's rate, because this stays on NVLink
if slow here: the problem is not the network. Check nvidia-smi nvlink -s for inactive links
and nvidia-smi topo -m for an unexpected layout.
2. two nodes, 16 GPUs
healthy: close to 50 GB/s per GPU
if this is the first drop: one pair's network path is bad. Go to step 3.
3. pairwise point to point
ib_write_bw between the two nodes, per NIC pair, message size 1 MB and above
healthy: about 45 to 48 GB/s on a 400 Gb/s port, and 1.5 to 2 us on ib_write_lat
a single pair well below that names a NIC, a cable or a switch port.
4. four, eight, sixty-four nodes
healthy: bus bandwidth stays flat as scale grows. A ring's per-rank traffic does not grow
with N, so a fabric with enough bisection bandwidth holds the number.
if the drop appears only here: congestion or ring layout, which is step 5.
5. the ring and the counters
NCCL_DEBUG=INFO shows the rings built and whether GDRDMA is in use
switch port counters: pause frames, ECN marks, discards, symbol errors, link flaps
healthy: pause and discard counters flat during the run
sanity: the whole sequence is five measurements and about fifteen minutes, against a job
losing a third of its throughput every hour it runs. Measure before touching a
setting, because every variable you change without a diagnosis has to be unchanged
later by someone who will not know why it was set
Debugging a Slow All-Reduce is the concept page; this is the drill with the numbers attached.
The four causes, and what each looks like:
| Cause | Signature | Fix |
|---|---|---|
| GPUDirect disabled | Roughly half the expected bandwidth from two nodes upward, GDRDMA absent in the NCCL log, higher CPU use | Load the peer-memory module; check the GPU and NIC are on the same PCIe root; set NCCL_NET_GDR_LEVEL |
| One bad link or port | The drop appears at the first scale that includes that node, and ib_write_bw names the pair | Replace the cable or transceiver; drain the node; check symbol error counters on the port |
| Wrong NIC per GPU | All pairs healthy, bandwidth still low, rings in the log hop between rails | Set NCCL_IB_HCA to bind each GPU to its nearest interface, or supply a topology file |
| Congestion | Only at scale, pause frame or ECN counters climbing on switch ports during the run | Fabric configuration, or the fabric is oversubscribed for this traffic pattern |
The one cause this sequence hides is intermittency. A link that flaps once an hour passes every check you run and destroys a long job, because a collective stalls until a timeout fires. That needs a different instrument: link counters sampled continuously and compared over days, plus the NCCL flight recorder enabled so that when a job does hang there is a record of which rank never arrived. Stragglers and Hangs covers that path, and the tell that sends you there is a job whose bandwidth is fine when measured and whose wall-clock progress is not.
The reversal condition: this whole procedure assumes the collective is actually slower than the fabric allows. Before running any of it, check that the expectation is right. A job doing an all-reduce of small tensors many times per step is latency-bound rather than bandwidth-bound, and its bus bandwidth will read low for a reason no cable replacement will fix. Compare the message size against the crossover where the ring's latency term stops dominating, and if the messages are small the answer is bucketing them, not repairing the fabric.
What interviewers probe next
- "Why bus bandwidth rather than algorithm bandwidth?" Algorithm bandwidth is message size over time and is not comparable across collectives or rank counts. Bus bandwidth applies the factor each collective actually moves, so it compares against line rate directly.
- "What does it mean if bandwidth falls as you add nodes?" Either the fabric is oversubscribed at that scale or the rings begin crossing the spine. A well-provisioned fat tree holds the number flat, since per-rank ring traffic does not grow with N.
- "How do you tell a bad cable from a bad NIC?" Move the cable to a known-good port and re-measure. Symbol error counters on the switch port also localize it, since a failing transceiver raises them on one side.
- "What if only one job is slow and others are fine?" That points at placement and rank mapping rather than at hardware, since the hardware is serving the other jobs.
Common mistakes
- Changing environment variables before measuring at any scale, which produces movement without a diagnosis.
- Reading algorithm bandwidth and comparing it against line rate.
- Testing on two nodes only, which never creates the congestion that appears at 64.
- Concluding the fabric is broken when the job's messages are too small to be bandwidth-bound at all.
Key takeaways
- Localize by scale first: 1, 2, 4, 8, 64 nodes, and the breaking point names the layer.
- Healthy numbers: about 45 to 48 GB/s on
ib_write_bwfor a 400 Gb/s port, bus bandwidth near 50 GB/s per GPU across nodes, and flat as scale grows. - Four causes with distinct signatures: GPUDirect off, one bad link, wrong NIC per GPU, congestion.
- Intermittent faults pass every check; they need continuous counters and the NCCL flight recorder instead.