Debugging a Slow All-Reduce

TL;DR: Reproduce the slowdown with the application's message sizes and placement, interpret the benchmark's units, then vary one part of the path. algbw measures logical payload per elapsed time; all-reduce busbw applies a rank-count normalization. Compare equivalent configurations and actual link counters before blaming a NIC, GPUDirect, or the fabric.

Begin with one collective

An all-reduce combines one array from each rank and returns the result to every rank. A rank is a participant, usually mapped to one GPU in this test. Four ranks contributing one number each might all receive their sum. The operation completes only when the required contributions and transfers finish.

A slow training step can therefore include late rank arrival, device execution, communication and poor overlap. NCCL timing alone does not separate them. For the vocabulary, NCCL and Collective Algorithms is readable without an account. This page teaches the diagnostic experiment rather than the collective implementation.

Suppose two nominally similar jobs report different bandwidth. Before changing a flag, save the rank-to-GPU mapping, nodes, message size, dtype, in-place mode, software versions, test iterations and timing options. Without those, the two numbers may describe different experiments.

Read algbw and busbw without changing units

NVIDIA's nccl-tests performance note defines algbw = S/t and, for all-reduce over n ranks, busbw = algbw × 2(n-1)/n. Here S is one rank's logical array size in bytes and t is seconds. The correction is tied to the collective, not proof that NCCL selected a ring.

For an invented eight-rank result with a 256 MiB array and 8 ms elapsed:

S = 256 × 1,048,576 = 268,435,456 bytes
t = 8 / 1,000 = 0.008 seconds
algbw = S / t / 1e9 = 33.554432 GB/s
all-reduce factor = 2 × (8 - 1) / 8 = 1.75
busbw = 33.554432 × 1.75 = 58.720256 GB/s
sanity: both columns come from the same 8 ms, not two measurements.

A nominal 400 Gb/s port corresponds to 50 GB/s before overhead. That does not make this result impossible: busbw is a normalized collective rate, not a counter of bytes traversing that port. GPU-local paths, multiple NICs, rank placement and collective algorithms determine the relevant physical comparison. Do not add transmit and receive rates to a one-direction limit without defining the boundary.

Check the printed time unit in your build. The test implementation also supports timing and rank-aggregation choices; a reported average does not give a latency percentile. Small messages emphasize overhead, while large messages can reveal a bandwidth plateau. Test both when the application uses both.

Separate the application from the transport

Run all_reduce_perf on the affected allocation with correctness checking enabled, representative message sizes, warm-up and repeated runs. Follow the nccl-tests launcher instructions for your MPI build and process layout. This page's arithmetic runs on a CPU; it does not claim a GPU benchmark was executed here.

If the isolated test matches a known-good baseline but the application remains slow, inspect rank arrival and compute/communication overlap in a timeline. The test has narrowed the hypothesis; it has not cleared every network path under concurrent production traffic.

If the isolated test is slow, compare within-node pairs, cross-node pairs and the full group. Keep software, GPU/NIC affinity and sizes fixed where possible. A bisection that makes both halves fast may have removed congestion or changed the algorithm. It does not prove every node is healthy. Restore scale to test that explanation.

Follow a symptom with a discriminating check

NVIDIA's network diagnostics describe ibstat for negotiated port state/rate and ib_write_bw for point-to-point tests. Record the tested memory type. A host-buffer test cannot demonstrate the GPU-to-NIC path. Take counter deltas during the same interval; a lifetime error total may predate the slowdown.

For GPU Direct RDMA, read the selected transport and memory-registration evidence from your installed NCCL stack. NCCL_DEBUG=INFO can help, but a single log substring is not a portable certification. NVIDIA's GPU troubleshooting guide documents both nvidia-peermem and a supported DMA-BUF path. A missing peer-memory module alone does not establish that direct access is disabled.

Use the actual GPU/NIC topology and container-visible devices to explain the selected path. On RoCE, rising pause counters warrant investigation alongside congestion marks, retries and receiver behavior; they do not prove congestion marking is absent. Check GPU clocks and competing work when transfers wait on a slow participant.

Know when a tuning change is a fix

Preserve the failing baseline, change one supported setting, then repeat the same experiment and the application workload. An algorithm override can be a legitimate workaround for a particular shape or software version. It is neither proof of a hardware fault nor a universal repair. NVIDIA's performance triage puts tuning after system checks.

Attach mappings, timing distributions, counter deltas and correctness results to an escalation. Avoid draining arbitrary nodes based on a bandwidth ratio alone. For deeper path analysis, Topology-Aware Communication is a premium continuation.

What interviewers are listening for

If the same 33.55 GB/s algbw came from four ranks, the factor would be 1.5 and busbw about 50.33 GB/s. The change in normalization is not a speedup. If only the full group is slow, propose a test for shared-link contention or collective selection before naming a broken NIC.

Key takeaways

• Eight-rank all-reduce multiplies algbw by 1.75; both describe the same elapsed interval.
• Compare equivalent runs first. A normalized rate is not a NIC byte counter.
• Host-memory tests and missing modules cannot, alone, establish the GPU Direct path.