mediumNewNVIDIAMeta

How long does an all-reduce of a 70B model's gradients take on eight GPUs?

The ring all-reduce formula, 2(n−1)/n times the buffer over the link bandwidth, applied to 141 GB of bf16 gradients on an NVLink node: 0.27 s by the formula, and the two reasons the measured number is different.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Advanced

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.

Foundational

🕸️ Distributed Training

Collective Communication PrimitivesAll-reduce, all-gather, reduce-scatter, all-to-all and broadcast are the five operations every parallelism strategy is built from, and each has a fixed per-rank traffic cost you can compute before a job runs. Knowing those volumes for a named model is how you decide whether a layout is compute-bound or waiting on the network.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow many tokens per second can one B200 decode for a 70B model?Next in this trackWhat storage throughput does a 16k-GPU training cluster need?Next in this trackTraffic peaks at three times the daily average. Capacity-plan the serving fleet.

FEDITOR'S NOTE

The interviewer wants the ring factor derived, the bytes and the bandwidth with units, a time, and then the sentence about overlap: 0.27 s only matters if it is exposed.

DISCUSSION · 0

No comments yet — be the first to share your approach.