Collective Communication Primitives

TL;DR: Five collectives cover distributed training: all-reduce (everyone ends with the sum), all-gather (everyone ends with everything), reduce-scatter (everyone ends with one summed shard), all-to-all (everyone sends a distinct piece to everyone) and broadcast. On a ring, all-gather and reduce-scatter each move (N-1)/N of the full buffer through every rank, all-reduce is the two back to back at 2(N-1)/N, and those factors are what turn a model size into seconds on a fabric.

The five operations

Picture N ranks, each holding either a full buffer of M bytes or a shard of M/N.

The row that decides most interviews: all-reduce is reduce-scatter followed by all-gather. The ring algorithm literally runs one then the other, and its 2(N-1)/N cost is the sum of the two (N-1)/N halves. ZeRO exists because someone noticed you can keep the halves apart. Reduce-scatter leaves each rank holding exactly the gradient shard its optimizer partition needs, and the all-gather can wait until the parameters are needed again.

The volumes for a 70B model

Llama 3.1 70B has 70.6B parameters, so the BF16 gradient (or the BF16 parameter set) is M = 141.2 GB. Per-rank traffic below means bytes each GPU must send, with the same amount received:

Going from 8 to 64 ranks adds only 12% to the per-rank bytes. Bandwidth-optimal collectives are set by M and the per-rank link speed; N enters only through the step count, which is the latency term covered in ring vs tree.

Now the fabric, as of 2026. NVLink on an H100 SXM is 900 GB/s bidirectional per GPU, 450 GB/s each way, so the 8-rank all-reduce needs at least 0.55 s at the spec ceiling. Across nodes at 400 Gb/s per GPU, roughly 50 GB/s, the 64-rank all-reduce needs 5.6 s and the FSDP step 8.3 s. Against a compute time of about 70 s per step (65,536 tokens per GPU, 6 × 70.6e9 × 65,536 FLOP at 40% of 989 TFLOPS), 8.3 s is 12% and hides behind compute with prefetch. Cut tokens per GPU to 8,192 and compute drops to 8.7 s while the collectives still take 8.3 s. Same model, same fabric, and the job is now network-bound. This calculation, done out loud, is what an interviewer means by "would this layout scale."

Reading the numbers NCCL gives you

nccl-tests reports two bandwidths and candidates mix them up. algbw is bytes in the buffer divided by time, the number an application feels. busbw rescales it by the collective's traffic factor (2(N-1)/N for all-reduce) so it can be compared directly with link bandwidth. An all_reduce_perf run on 8 H100s with a busbw far below what NVLink allows means a topology or configuration problem, not a slow model. A busbw near the link rate alongside a poor training step means the application is issuing collectives too small or too serialized to reach it.

The environment variables an operator actually reads: NCCL_DEBUG=INFO prints the rings and trees NCCL built and which NIC each rank chose. NCCL_ALGO and NCCL_PROTO override the algorithm and protocol for a bisection test, never for production tuning. NCCL_IB_HCA and NCCL_SOCKET_IFNAME pin the interfaces when a node has a management NIC that NCCL should never touch. NCCL_NET_GDR_LEVEL controls whether GPUDirect RDMA is used for a given GPU-to-NIC distance.

Where it breaks in production

Collectives are barriers in disguise. Every rank must arrive, so the slowest rank sets the pace of the step, and a single GPU at reduced clocks or one NIC with CRC errors degrades the whole job. A hang inside a collective is indistinguishable from a slow one until a timeout fires, which is why the flight recorder (TORCH_NCCL_TRACE_BUFFER_SIZE) exists: it records which rank never entered which collective.

Small collectives are the other trap. A 134 MB tensor-parallel all-reduce is latency-sensitive in a way a 141 GB gradient reduction never is, and it runs on the critical path of every layer. That is why tensor parallelism lives on NVLink and data parallelism can tolerate InfiniBand.

What interviewers are listening for

They want the factor. "All-reduce costs 2(N-1)/N of the buffer per rank" is the sentence, followed by the decomposition into reduce-scatter and all-gather. The follow-up they hold back is "so why does FSDP cost 1.5x DDP?" Two all-gathers plus one reduce-scatter is three (N-1)/N halves against DDP's two. A candidate who says "FSDP reduces communication" has confused memory with traffic, and that usually ends the deep-dive.

Common misconceptions

• All-reduce is one primitive. It is two, and treating it as one hides the trick ZeRO is built on.
• More ranks means proportionally more communication per rank. Per-rank bytes saturate near 2M; only the latency term keeps growing.
• busbw is what your training loop gets. That is algbw; busbw is a normalized figure for comparing against link speed.
• Broadcast is cheap because only one rank sends. On a ring every rank forwards, and it costs the same (N-1)/N as an all-gather.

Key takeaways

• All-gather and reduce-scatter cost (N-1)/N × M per rank; all-reduce is both, 2(N-1)/N; all-to-all is (N-1)/N of the per-rank buffer.
• For Llama 3.1 70B in BF16, a 64-rank all-reduce is 278 GB per rank, about 5.6 s at 50 GB/s.
• Per-rank volume barely changes with N; tokens per GPU per step decides whether it hides behind compute.
• Compare busbw with link speed and algbw with your step time.