NCCL and Collective Algorithms
NCCL implements GPU collectives using algorithms, protocols and channels over available hardware paths. Learn what each choice changes, how to read bandwidth results, and which evidence separates a communication problem from late application work.
TL;DR: NCCL (NVIDIA Collective Communications Library) implements collective communication for NVIDIA GPUs. The operation specifies the result; algorithms, protocols and channels determine how to move and combine the data. More channels cannot create network bandwidth. Diagnose with a matched benchmark, topology and timing evidence before forcing a tuning choice.
One operation, several execution choices
Four training processes each hold a gradient vector. An all-reduce sum must leave the summed vector on every participant. A rank identifies one participant; a communicator defines the participating group. They must agree on the operation, count, dtype and collective ordering. The Collective Communication page follows that ownership contract.
A framework selects a communication backend. NCCL is the recommended PyTorch backend for CUDA collectives; CPU groups can use Gloo. See the PyTorch backend guide.
Across hosts, the bytes pass through a network interface card (NIC). The table separates the execution choices from the physical resources they use.
| Choice | What it controls | What it does not establish |
|---|---|---|
| Algorithm | Who exchanges and combines which chunks, such as ring or tree | That this schedule is fastest for every size |
| Protocol | How chunks and synchronization signals move, such as LL, LL128 or Simple (two low-latency modes that trade some bandwidth for fast small messages, and a high-throughput mode for large ones) | That a protocol is usable on every platform |
| Channels and work allocation | How communication work is split across GPU execution resources | One independent NIC per channel |
| Transport and placement | Which GPU links, network paths and interfaces carry the work | That detected links reach their advertised rate |
NCCL considers topology and message characteristics when selecting execution. Small-message startup and large-message throughput favor different choices, but there is no universal size where tree beats ring. Eligible hardware can also provide reduction offload. Increasing communication work on the GPU can contend with compute, so a faster isolated collective need not make a faster training step. NVIDIA's tuning explanation develops that tradeoff.
Two channels still share one NIC
Consider one 64 MiB transfer, not a complete all-reduce. Assume its only bottleneck is a shared NIC delivering a usable 16 GiB/s in the sending direction. Splitting the transfer into two 32 MiB streams leaves the same 64 MiB crossing that NIC:
64 MiB ÷ 16 GiB/s = 0.00390625 s = 3.90625 ms.
Two independent 16 GiB/s paths could each carry 32 MiB in 1.953125 ms, if the endpoints and rest of the path sustain both concurrently. That improvement requires more usable physical capacity. Extra channels can help fill an underused path, but cannot double a saturated link. These are calculated payload floors, excluding startup, synchronization and reduction work; they are not benchmark results.
Read both bandwidth columns
Suppose a four-rank all-reduce of 64 MiB per rank takes an illustrative 6 ms. Use bytes and seconds consistently:
| Quantity | Calculation | Result |
|---|---|---|
Algorithm bandwidth, algbw | 67,108,864 / 0.006 bytes/s | 11.18 GB/s |
| All-reduce normalization | 2 × (4−1) / 4 | 1.5 |
Normalized bus bandwidth, busbw | 11.1848107 × 1.5 | 16.78 GB/s |
GB/s here uses decimal bytes, while MiB uses powers of two. The nccl-tests definition normalizes by operation and group size. busbw is not a measured NIC counter, proof that a ring ran, or automatically comparable to a bidirectional hardware headline. Record elapsed time, correctness and both bandwidth columns. Identify the physical boundary before comparing rates.
Check yourself: keep 64 MiB and 6 ms, but change the group to eight ranks. algbw stays 11.18 GB/s; the factor becomes 1.75, so busbw rises to 19.57 GB/s. The operation did not become faster. The reporting normalization changed.
Read logs as evidence, with their limits
For an investigation, INFO logging with INIT,NET,GRAPH,TUNING subsystems can expose initialization, network selection, topology and tuning information. Give each process its own debug file using host/PID placeholders such as %h.%p, and a separate prefix for each run. INFO output is not an exhaustive trace of every collective's chosen execution. The NCCL environment reference defines these settings and their version limits.
Three distinctions prevent misleading diagnoses:
NCCL_SOCKET_IFNAMEselects IP interfaces for socket communication, not just bootstrap.NCCL_IB_HCAselects RDMA interfaces/ports;=mlx5_0:1is an exact match, while a name prefix can include additional devices.NET/IBnames the verbs transport; the physical link may be InfiniBand or RoCE. Check link-layer and negotiated-rate information rather than inferring them from that label. See network troubleshooting.- Missing a particular log suffix or the
nvidia-peermemmodule does not prove host-memory staging. Supported systems may use DMA-BUF. Check the actual GPU-memory path and platform prerequisites in GPU troubleshooting.
Protocol overrides are experiments. LL128 is not an NVLink-only protocol, and forcing it on an unsupported platform can corrupt data. Keep supported defaults unless a controlled test justifies a change; retain correctness checks.
Decide what to investigate next
Start with a benchmark matching the application's group, placement, message sizes and dtype. A healthy isolated result plus long application collectives points toward arrival, scheduling or workload effects. A slow matched result calls for path, topology, configuration and algorithm investigation. Neither pattern proves a single cause. A two-node result below a one-node result is not enough to rule out algorithm selection.
Measure GPU completion, not just the host call's return: NCCL work is enqueued asynchronously on CUDA streams. Preserve comparable compute overlap when judging end-to-end impact.
In an interview, say which result would change your next step: “If the matched collective remains slow with equivalent arrivals, I will investigate communication. If it recovers, I will trace the preceding dependencies.” Debugging a Slow All-Reduce develops that investigation. The Premium late-rank lesson works through two identical step times with different causes.
Key takeaways
- Separate operation, algorithm, protocol, channels and physical transport.
- Two streams sharing a saturated 16 GiB/s NIC still need at least 3.90625 ms for 64 MiB.
- At four ranks, 64 MiB in 6 ms means 11.18 GB/s
algbwand 16.78 GB/sbusbw; neither is a NIC counter. - Keep a tuning change only when correctness and representative full-step measurements support it; retest after environment changes.