TL;DR: Each GPU holds a full copy of the model and trains on its own slice of the batch. What crosses the wire is the gradient, all-reduced once per step, so every rank ends the step with the same averaged gradient and applies the same update. A ring all-reduce moves 2(n−1)/n × (gradient bytes) through each GPU, which is about twice the model size in bf16 for any n above a handful, and DDP launches it bucket by bucket during the backward pass so most of it is hidden behind compute.
How to approach it
Start by saying what is replicated and what is split: weights and optimizer state replicated, batch split. Then name the one tensor that has to move, the gradient, and say why: every replica must apply the same update or the copies diverge. Write the all-reduce cost formula next, with n and the gradient size as named variables, and compute it for a concrete model before comparing it to the compute time of the step. Close by saying when it happens, during backward, not after, because that is the detail that separates a candidate who has read the DDP source from one who has read a blog.
A strong answer
A typical situation: a team trains Llama 3.1 8B on 64 H100s with plain Data Parallelism and DDP. Every GPU has the same 8.03 B parameters, the same Adam state, and a different 8k-token micro-batch. After the backward pass each GPU has a local gradient that reflects only its own data. The correct gradient for the global batch is the mean of the 64 local gradients, and computing that mean and delivering it to every rank is the only communication the algorithm needs.
Three things do not move. Input data goes from storage straight to the rank that consumes it. Weights never move, because every rank already has them and applies an identical update. Activations never move, because each rank runs its own complete forward and backward.
The gradient is the same size as the model in whatever precision it is kept. In bf16 that is 2 bytes per parameter.
inputs: N = 8.03e9 parameters, gradient dtype bf16 = 2 B/param
n = 64 GPUs, one 400 Gbps NIC per GPU ≈ 50 GB/s
gradient bytes G = N × 2 B = 16.1e9 B ≈ 16 GB
ring all-reduce per GPU = 2 × (n − 1)/n × G
= 2 × 63/64 × 16 GB
= 31.6 GB sent (and the same received) per GPU per step
time at 50 GB/s = 31.6 GB ÷ 50 GB/s ≈ 0.63 s
sanity: 2(n−1)/n tends to 2, so the per-GPU traffic is "twice the model" for any n
above about 8, and it does not grow with more GPUs; the ring is bandwidth-optimal.
The factor of 2(n−1)/n comes from the two phases of the ring. Reduce-scatter sends (n−1)/n of the buffer around the ring so that each rank ends up owning the fully reduced sum of one 1/n slice, then all-gather sends the same amount again so every rank collects every slice. Each phase moves (n−1)/n × G per rank, and the two together give the formula. Collective Communication Primitives covers the other collectives; for data parallelism the all-reduce is the one that matters.
Now compare that 0.63 s to the compute in the step:
compute per GPU per step = 6 × N × tokens per GPU
= 6 × 8.03e9 × 8,192 ≈ 3.95e14 FLOPs
time at 989 TFLOPS × MFU 0.4 = 3.95e14 ÷ 3.96e14 ≈ 1.0 s
communication ÷ compute ≈ 0.63 ÷ 1.0 = 63%
If the all-reduce ran after the backward pass finished, this run would spend about 40% of its wall clock waiting on the network. It does not, because DDP registers a hook on every parameter and fires the all-reduce for a bucket of gradients (25 MB by default, bucket_cap_mb) as soon as the backward pass has produced them. The backward pass computes the last layer's gradient first, so the last layer's bucket is on the wire while the earlier layers are still being differentiated. In the best case only the first layer's bucket is exposed. The overlap is the reason data parallelism scales at all, and it is also why the achievable batch per GPU and the network bandwidth are coupled: shrink the micro-batch and the compute time falls while the gradient bytes stay fixed.
Per-step traffic on NVLink inside one node is the same formula with a different link. At n = 8 over 900 GB/s the same 16 GB gradient costs 2 × 7/8 × 16 GB = 28 GB per GPU, about 31 ms, which is why an 8-GPU DDP run rarely notices its network and a 64-GPU run does.
The decision this leads to: plain data parallelism is the right first choice whenever the model plus its 16 bytes per parameter of training state fits on one GPU and the per-GPU compute per step is several times the all-reduce time. The condition that reverses it is memory. An 8B model already needs 128 GB of static training state, more than an 80 GB H100, so even this "small" example is in practice run with ZeRO and FSDP sharding the optimizer state, which changes the collective from an all-reduce to a reduce-scatter plus an all-gather of the same total bytes.
The reversal condition: a model whose 16 bytes per parameter no longer fit on one card. At that point plain data parallelism is not an option at all and the question becomes which axis to shard first, which ZeRO and FSDP answers. NCCL_DEBUG=INFO at startup confirms the ring the library actually built.
What interviewers probe next
- "Why the mean and not the sum?" The loss is a mean over the global batch, so the gradient of that loss is the mean of the per-rank gradients; NCCL sums, and the framework divides by world size (or scales the loss) before or after.
- "What if one rank's gradient is NaN?" The all-reduce sums it into every rank, so every replica gets a NaN update in the same step; the fix is a gradient-norm check before the optimizer step, on every rank, with a collective vote to skip.
- "Does gradient accumulation change the traffic?" It divides it: with k micro-batches per optimizer step, DDP under
no_sync()all-reduces once per k backward passes, so the bytes per step are unchanged but the bytes per token fall by k. - "How does the byte count change with fp32 gradients?" It doubles to 32 GB, which is why bf16 gradient communication with an fp32 master copy is the default, and why some stacks reduce in fp32 only for the final accumulation.
Common mistakes
- Saying the weights are broadcast every step. They are broadcast once, at initialization, and never again in a healthy run.
- Computing the all-reduce cost as n × model size, which is the naive all-to-one reduction, not a ring.
- Forgetting the per-direction detail: NIC and NVLink figures are usually quoted bidirectional, and the ring sends and receives at the same time, so the 50 GB/s and 900 GB/s figures used here are the generous case.
- Treating the all-reduce as a serial phase after backward, and then concluding data parallelism cannot work at 64 GPUs.
Key takeaways
- Gradients move; weights, optimizer state and data do not.
- Per-GPU traffic = 2(n−1)/n × gradient bytes: about 32 GB per step for an 8B model in bf16, independent of n once n is past a handful.
- At 50 GB/s per NIC that is about 0.6 s against about 1 s of compute per 8k-token micro-batch, so the overlap with backward is what makes it viable.
- Data parallelism is the default until the 16 bytes per parameter of training state stops fitting on one card, which for an 8B model is already the case.