Collective Communication Primitives
Follow small vectors through all-reduce, reduce-scatter, all-gather and all-to-all. Separate the result each worker needs from the algorithm-specific byte count and measured completion time.
TL;DR: A collective specifies what a group of workers should hold after communication. All-reduce gives everyone a combined result; reduce-scatter gives each worker one combined shard; all-gather assembles shards everywhere. Those are data-placement rules. The algorithm and topology determine the traffic and time.
Follow four numbers
A rank is one worker in the communication group, usually one process driving one GPU. Two ranks begin with vectors: rank 0 holds [1, 2], and rank 1 holds [10, 20]. Choose summation as the reduction operation.
| Operation | Result |
|---|---|
| All-reduce | Both ranks receive [11, 22] |
| Reduce to rank 0 | Rank 0 receives [11, 22]; rank 1 has no promised reduced output |
| Reduce-scatter | Rank 0 receives [11]; rank 1 receives [22] |
| All-gather of those shards | Both ranks receive [11, 22] |
| Broadcast from rank 0's original vector | Both ranks receive [1, 2] |
Reduce-scatter followed by all-gather produces the same result as all-reduce here. That decomposition explains a ring all-reduce, but the public all-reduce operation does not require every implementation to execute those two literal phases. A reduce followed by broadcast also produces the result.
Notice what all-gather does not do: it does not add values. It concatenates contributions in rank order. If two ranks independently all-gather [1, 2] and [10, 20], each gets [1, 2, 10, 20], not [11, 22].
All-to-all changes the destination
All-to-all exchanges different pieces with different ranks. Label a piece by its source and destination:
| Starting rank | After the exchange |
|---|---|
| Rank 0 sends A to 0, B to 1 | Rank 0 receives A from 0 and C from 1 |
| Rank 1 sends C to 0, D to 1 | Rank 1 receives B from 0 and D from 1 |
This is useful when routing tokens to experts: a token goes to the rank that owns the selected expert. There is no summation in the exchange itself. Real routing can be imbalanced; equal-sized all-to-all is only one communication layout, and variable-size routing needs counts, padding or another exchange scheme.
Separate gradient ownership from parameter ownership
Data Parallelism and DDP wants the complete reduced gradient on every replica. A sharded optimizer can instead consume one reduced gradient shard per rank. After its local update, a rank holds an updated parameter shard. Gathering those parameter shards reconstructs parameters, not gradients.
Keeping these tensor names attached to arrows prevents a common mistake: drawing an optimizer between reduce-scatter and all-gather and calling the whole path a gradient all-reduce. The update changed the data. ZeRO and FSDP uses these primitives with schedules that depend on which state is sharded and when full parameters are released.
A byte count you can reproduce
Take four ranks and a full buffer of 128 MiB. Divide it into four 32 MiB shards. In a conventional ring, each rank sends one shard in each of three rounds for reduce-scatter, then does the same for all-gather.
| Ring operation | Bytes sent per rank |
|---|---|
| Reduce-scatter | 3 × 32 = 96 MiB |
| All-gather | 3 × 32 = 96 MiB |
| All-reduce | 96 + 96 = 192 MiB |
Each rank receives the same volume. Do not add sending and receiving and then divide by a one-direction bandwidth when the link can do both concurrently. For N ranks and a full M-byte buffer, these ring send factors are (N−1)/N and 2(N−1)/N.
Those factors are not a universal cost table for every collective. A chain broadcast, for example, forwards the entire M-byte buffer from the root through intermediate ranks; it does not give every rank the ring all-gather cost. Algorithm, message partitioning and topology matter.
At an assumed usable send rate of 16 GiB/s, 192 MiB takes 192 / 16,384 = 0.01171875 s, about 11.72 ms, before startup costs and other delays. This is a bandwidth-only estimate. A late rank, shared NIC or congested path can increase completion time. The buffer example makes no claim about whether a particular training model fits in memory.
Reading a benchmark without overclaiming
In nccl-tests, algorithm bandwidth relates the operation's buffer size to elapsed time. Bus bandwidth applies a collective-dependent normalization; for all-reduce it multiplies by 2(N−1)/N. It is useful for comparisons under a stated algorithm/model, but it is not a measurement of every physical link's utilization.
A small-message result below peak does not prove bad wiring. Startup overhead may dominate. Compare matching rank placement, message sizes, dtype and topology, then inspect the application timeline. A fast isolated collective does not prove it is hidden behind training compute.
Where execution goes wrong
Participating ranks must issue compatible collectives in matching order, with the required counts, datatypes and roots. If one rank skips a call, peers can hang or fail. A collective is not a general-purpose barrier: asynchronous submission and local completion have specific stream and dependency rules. Wait for the result before consuming it, using the framework's documented synchronization semantics.
Start a diagnosis with the first mismatched or late operation, not the last rank to time out. Record group membership, operation sequence and tensor shape alongside timings. That separates a programming mismatch from a slow transport.
Check your understanding
Can all-gather replace an all-reduce? Only if the application also performs the missing reduction. Does a 192 MiB ring send count mean the model has 192 MiB of gradients? No: our full input buffer was 128 MiB. Does sharding always reduce communication? No: retaining less state can require additional gathers later.
Key takeaways
- Reduce-scatter changes ownership; all-gather assembles shards without adding them.
- A four-rank ring all-reduce of 128 MiB sends 192 MiB per rank.
- The 11.72 ms bandwidth estimate assumes 16 GiB/s and excludes startup and waiting.
- Choose the output placement first, then verify the algorithm and application timeline.