NCCL and Collective Algorithms

TL;DR: NCCL turns one all_reduce call into a set of rings or trees laid over the NVLink and NIC topology it detected at init, split across channels so every link carries traffic in parallel. Every one of those decisions is printed by NCCL_DEBUG=INFO and can be overridden by an environment variable, and nccl-tests is how you find out whether the decision reached the bandwidth the hardware allows.

What happens before the first byte moves

When a process group is created, each rank builds a picture of its own machine: which GPUs sit on which PCIe switch, which pairs share NVLink, which NICs are closest to which GPU. NCCL reads this from sysfs and NVML, then runs a graph search for the rings and trees that use the most bandwidth without crossing a slow hop twice. Only then does it open connections, and it tells you exactly what it chose:

NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_3:1/IB ... [7]mlx5_9:1/IB
NCCL INFO Using network IB
NCCL INFO Channel 00/16 : 0 1 2 3 4 5 6 7
NCCL INFO Channel 00/0 : 7[7] -> 8[0] [send] via NET/IB/7/GDRDMA
NCCL INFO Connected all rings
NCCL INFO Connected all trees

Three words in that output decide performance. IB versus Socket tells you whether RDMA is in use at all. GDRDMA versus a bare NET/IB/7 tells you whether the NIC reads GPU memory directly or bounces through host RAM. And Channel 00/16 tells you how many parallel streams the message will be cut into.

Rings, trees, channels and protocols

A ring all-reduce is a reduce-scatter followed by an all-gather. Over n ranks it takes 2(n-1) steps and each rank sends and receives 2(n-1)/n times the buffer size, which is bandwidth-optimal and why NCCL uses it for large messages. Its latency grows linearly with n, so at thousands of ranks a 1 MB all-reduce spends its time waiting on hops. NCCL's double binary tree has log n depth and takes over for small buffers and large rank counts. NCCL_ALGO forces one or the other; leave it alone unless a measurement says the cost model got it wrong.

Channels are the part people miss. NCCL does not run one ring. It runs up to NCCL_MAX_NCHANNELS rings in parallel, each with its own CUDA thread blocks and its own path through the NICs. On an 8-NIC node you need enough channels that every NIC is busy at once, or you are paying for 400 GB/s of NICs and using 50. Protocols are the inner loop: Simple for large messages, LL (an 8-byte flag packed with each 8 bytes of data, cheap for tiny buffers) and LL128 (128-byte lines, only on NVLink). NCCL_PROTO overrides.

Two hardware-assisted paths exist as of 2026: NVLS uses NVSwitch's in-network reduction on Hopper and Blackwell nodes (NCCL_NVLS_ENABLE), and CollNet uses SHARP on Quantum InfiniBand switches (NCCL_COLLNET_ENABLE). Both cut the number of times data crosses a link.

The numbers an interviewer expects

nccl-tests reports two bandwidths. algbw is bytes divided by time. busbw multiplies that by 2(n-1)/n for all-reduce so it can be compared to link speed. Always quote busbw.

Worked example, assumptions stated. Llama 3.1 70B (70.6B parameters) trained with tensor parallel 8 inside each H100 node and data parallel across 64 nodes over a rail-optimized 400 Gb/s fabric. Each GPU's gradient shard in BF16 is 70.6e9 / 8 x 2 B, roughly 17.7 GB. A ring across 64 data-parallel peers moves 2 x 63/64 x 17.7 GB, about 34.8 GB per rank per step. One 400 Gb/s NIC per GPU is 50 GB/s per direction, so line rate is roughly 0.7 s. Nobody hits line rate; at 80% of it, expect close to 0.9 s. If the backward pass takes 2 s, DDP-style bucketed overlap hides most of that. Reduce in FP32 and the number doubles. That last sentence is the follow-up.

Inside the node the numbers are different by an order of magnitude: H100 NVLink is 900 GB/s bidirectional per GPU, so 450 GB/s per direction, nine times the NIC. Every parallelism decision in the training track flows from that ratio.

The variables that change a decision

Running the test that settles arguments

# one node, 8 GPUs, 8 B to 8 GB doubling each step
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

# 16 nodes, one process per GPU, pinned to the compute rails
mpirun -np 128 -N 8 -x NCCL_DEBUG=INFO -x NCCL_IB_HCA=mlx5 \
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Read the last few rows. Large-message busbw on a single H100 node should sit at a large fraction of the 450 GB/s NVLink per-direction figure; across nodes it should approach the 50 GB/s of one NIC per GPU. A two-node result far below the one-node result is a fabric or GDR problem, never a NCCL algorithm problem.

Where it breaks in production

The common failures are boring and repeat. A container ships a NCCL build that does not match the host's driver or the IB user-space libraries, and NCCL falls back to NET/Socket at a fortieth of the speed. A node loses nvidia-peermem after a kernel update and the log silently loses GDRDMA. Someone sets NCCL_ALGO=Ring in a shared launch script years ago and every 4 KB all-reduce on 2,048 ranks now walks 4,094 hops. A storage NIC gets enumerated as mlx5_2 and NCCL cheerfully includes it, so one rail's traffic goes over a 100 Gb/s port shared with checkpoint writes.

What interviewers are listening for

The held-back follow-up is: "busbw is 95% of line rate on one node and 45% on two. What do you check first?" They want NCCL_DEBUG=INFO for GDRDMA, then NCCL_IB_HCA to confirm the rail NICs, then ib_write_bw between the two nodes to separate NCCL from the link. Candidates who answer "raise NCCL_BUFFSIZE" or "switch to tree" are guessing at knobs, and the interviewer knows it.

Common misconceptions

• NCCL always picks the best algorithm. It picks from a cost model with tuned constants; on an unusual topology or a new NIC the constants are wrong and the log is how you find out.
• algbw is the bandwidth. Compare busbw to the link, or an 8-rank result looks 25% slower than it is.
• More channels always help. Each channel costs SMs that the compute kernels wanted, and past the point where the NICs are saturated it only adds contention.

Key takeaways

• NCCL discovers topology at init, lays rings and trees over it, and splits every collective across channels; NCCL_DEBUG=INFO shows all three.
• Ring is bandwidth-optimal at 2(n-1)/n bytes per rank; tree wins on latency for small messages and large rank counts.
• Quote busbw, and compare it to 450 GB/s per direction on H100 NVLink and 50 GB/s per 400 Gb/s NIC.
• Most production NCCL problems are the wrong NIC, a lost GDRDMA, or a stale forced NCCL_ALGO, not the algorithm itself.