Debugging a Slow All-Reduce
A training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.
TL;DR: Isolate before tuning. Run
all_reduce_perffrom nccl-tests on the job's own nodes and readbusbw: it should approach the link rate (roughly 45 GB/s per GPU across nodes on 400 Gb/s, several hundred GB/s inside an NVSwitched node). If it is low, bisect: halves of the job, then pairs, until one node or link is the outlier. Then check in this order: is GPUDirect RDMA on (NCCL_DEBUG=INFOshowsGDRDMA); did NCCL build rings along the rails; is every NIC negotiated at full rate; are PFC pause counters climbing; is one rank throttled or on the wrong NUMA node. Change an environment variable only after the measurement says which layer is wrong.
Read the number first
nccl-tests prints two bandwidths. algbw is bytes divided by time from the caller's view; busbw corrects for the algorithm (for a ring all-reduce it multiplies by 2(N−1)/N) so it can be compared with the hardware link rate directly. Compare busbw at a large message size (1 GB is typical) with what the path should give:
| Path | Expected busbw, healthy |
|---|---|
| 8 GPUs, one NVSwitched H100 node | 400 to 480 GB/s (NVLS makes it higher than a plain ring's math) |
| 2 or more nodes, 400 Gb/s per GPU, rail-optimized | 40 to 48 GB/s per GPU |
| same, GPUDirect off (bounce through host) | roughly 20 to 25 GB/s |
| a TP group accidentally spanning two nodes | limited by the NIC, so tens of GB/s where NVLink should give hundreds |
A number a third of the expected value with no errors is a healthy cluster with something misplaced or misconfigured; a number that varies run to run is congestion or a straggler; a number that is correct at small scale and wrong at large scale is topology.
The isolation order
Step 1, measure in isolation. The training job's own all-reduce timing is polluted by overlap and by the job's placement. all_reduce_perf -b 8 -e 1G -f 2 -g 1 under mpirun (or the container's launcher) across exactly the job's nodes, one process per GPU, gives a clean number in a minute.
Step 2, bisect. If 64 nodes are slow, run 32 and 32. One half is usually fine. Halve again. Down at pairs, ib_write_bw between two NICs measures a single link without NCCL in the way. A single bad link, NIC or node found this way is the most common ending, and the whole bisection is under an hour.
Step 3, the node checks, once you have a suspect:
| Check | Command | What "bad" looks like |
|---|---|---|
| NIC link rate | ibstat / ibstatus | a port at 200 Gb/s (or worse) when its peers show 400; a cable or a transceiver |
| GPUDirect RDMA | NCCL_DEBUG=INFO startup lines; lsmod | grep peermem | transport shows NET/IB without GDRDMA; module missing after a driver update |
| NIC to GPU topology | nvidia-smi topo -m | the NIC for GPU i shows SYS (other socket) instead of PIX/PXB |
| Link errors | perfquery counters, ibdiagnet; NIC error counters on Ethernet | symbol errors, link downs, retransmits climbing |
| GPU health | nvidia-smi -q -d CLOCK,PERFORMANCE, DCGM | clocks throttled, an XID in dmesg |
| CPU affinity | the launcher's binding; numactl -H | ranks pinned to the socket far from their NIC |
Step 4, the fabric checks, when nothing is uniformly wrong with one node:
- Rings on rails:
NCCL_DEBUG=INFOprints each ring; a ring that hops GPU 0 → GPU 3 across nodes is crossing rails and using the spine. Fix the rank placement or the launcher's local-rank mapping (topology-aware communication). - PFC and ECN: pause-frame counters on the leaf ports climbing during the test means the fabric is pausing instead of marking (congestion control). ECN thresholds or a slow receiver.
- Spine collisions: a few uplinks saturated while most idle points to hash collisions; adaptive routing or spraying.
- Environment:
NCCL_IB_HCArestricting ranks to the wrong NICs,NCCL_NET_GDR_LEVELtoo strict, or aNCCL_ALGO/NCCL_PROTOoverride left behind from an earlier experiment. Unset overrides before drawing conclusions.
The cases that recur
A NIC negotiated at half rate after a cable reseat: everything looks healthy in the dashboards because "link up" is green, and the bisection lands on one pair. A driver upgrade that removed the peer-memory module: GDR off fleet-wide, every all-reduce at half, no errors. A scheduler change that stopped guaranteeing whole nodes: TP groups spanning nodes, step time 5x. A new switch firmware with different ECN defaults: pause counters through the roof on one rail. A single GPU thermally throttled: the collective waits for it, and the fabric is blamed. Each has a different fix and the same first step.
Working it in the room
This is a reported interview question at the GPU clouds ("debug a slow all-reduce" at Crusoe and others) and the scored answer is the isolation order, stated as a procedure with the commands and the expected numbers. The follow-up held back is "you tuned NCCL_ALGO and it got faster; are you done?" No: a healthy fabric does not need the override, and the underlying fault is still there for the next job. The answer that sounds right and fails is "restart the job on different nodes": it works today and teaches the fleet nothing.
What to remember
- Measure with
all_reduce_perfand comparebusbwwith the link rate: ~45 GB/s per GPU across 400 Gb/s nodes, hundreds inside a node. - Bisect halves to pairs, then
ib_write_bwon the pair; most cases end at one link, NIC or node. - Node checks: NIC rate, GDR on, NIC-GPU topology, link counters, throttling, affinity. Fabric checks: rings on rails, PFC pauses, spine collisions, leftover env overrides.
- A number at half with no errors is GDR off or a bounce through the host; at a third or worse with clean links, look at placement.
- Drain the node with the evidence attached; an override that makes it faster is a symptom, not a fix.