How GPUs add up their gradients

In data-parallel training, every GPU holds a full copy of the model and trains on its own slice of the batch. After the backward pass, each GPU has a gradient: one number per weight, saying how that weight should change. The gradients differ, because the GPUs saw different examples.

Before anyone updates a weight, every GPU needs the same thing: the sum of all the gradients (or their average, which is the sum divided by the number of GPUs). If the GPUs updated with their own gradients alone, the copies of the model would drift apart. The operation that leaves the sum on every GPU is called an all-reduce, and it runs on every training step.

The vectors below are real: each GPU starts with its own numbers, and every later stage adds them up for real, message by message. The target row is the sum each GPU must end up with.

The obvious method: every GPU sends its whole gradient to one GPU, which adds them up and sends the total back to everyone. It is correct, and it does not scale. That one GPU must receive a full gradient from every other GPU and send a full gradient to each of them, all through its own links.

For a 7-billion-parameter model, the BF16 gradient is 14 GB. On eight H100s in one server, collecting at one GPU takes 0.44 seconds at the NVLink rate; on 64 GPUs across eight servers it takes 31.8 seconds, because most of the gradients now cross the network into one GPU's network card. A training step cannot wait that long.

The ring all-reduce, used by NCCL and most training stacks, does the same job on 64 GPUs in 0.55 seconds, and that time barely grows as you add GPUs. The chart compares them on the same links.

Try it. Add GPUs and watch which method's time grows.

Further reading: Collective communication, Data parallelism and DDP.

Step through the ring on real numbers: each GPU splits its gradient into one chunk per GPU, passes a chunk to its neighbour and adds the one it receives, until every GPU owns the complete sum of one chunk.

Watch the finished sums travel the ring until every GPU holds the whole summed gradient, and check that the result equals the target exactly.

Count what each GPU actually sends as the ring grows, see why the total approaches two gradients however many GPUs join, and learn to read NCCL's bus bandwidth and algorithm bandwidth.

Shrink the message until start-up latency dominates, and compare the ring's 2(n − 1) steps with a tree's steps that grow with the logarithm of the GPU count.

Run a flat ring and a hierarchical all-reduce across servers, and count exactly how many bytes each one sends over the slow network.

Fill gradient buckets as the backward pass produces them, as PyTorch DDP does, and find the bucket size that leaves the least communication exposed.

Put in a real model, GPU count and network, and get the all-reduce time per step, the bus bandwidth, and the share of a training step it costs.