TL;DR: Ordinary DDP synchronizes gradients, grouped into buckets, so model replicas can apply the same update after processing different data. Default initialization also synchronizes parameters and buffers, and runtime buffer broadcasts may occur. For a ring all-reduce, each rank sends
2(n−1)/n × gradient bytesand receives the same amount. That is a traffic count, not a guarantee of overlap or step time.
How to approach it
Name the replicated state and the split batch first. Explain why gradients must agree, then count bytes using the actual communicated dtype. Finish with two checks: whether a full training replica fits, and where reduction completion sits on the backward timeline.
A strong answer
Imagine a hypothetical one-billion-parameter model trained across eight GPUs with plain Data Parallelism and DDP. Each rank stores the full model and its own optimizer state. The input pipeline gives each rank different examples; DDP itself does not shard those inputs.
Each replica's backward pass produces gradients from its own examples. If every rank has equal weight in the loss, averaging those gradients gives the gradient of the combined mean loss. Starting from identical weights and optimizer state, the replicas can then apply the same update locally. They stay aligned because they use the same reduced gradient and update rule, not because one worker sends everyone a new model after each step.
| What moves? | Scope |
|---|---|
| Gradients | Reduced during synchronized backward passes, usually in multiple buckets (groups of gradients reduced together, so communication can start before backward finishes) |
| Parameters | Synchronized at default DDP initialization; ordinary updates happen locally |
| Module buffers | Runtime broadcasts can synchronize buffers such as running statistics |
| Activations | Remain local in ordinary DDP; other parallelism schemes or layers can add communication |
| Training examples | Delivered by the input pipeline, which is separate from gradient synchronization |
Suppose the communicated gradients are FP32. That assumption must come from the training recipe or trace: a BF16 autocast forward does not automatically make stored gradients BF16.
parameters = 1,000,000,000
gradient bytes = parameters × 4 = 4 GB
ranks = 8
ring send factor = 2 × 7/8 = 1.75
sent per rank = 1.75 × 4 = 7 GB
received per rank = 7 GB
These are decimal GB. The ring splits the buffer into eight chunks. Reduce-scatter sends seven chunks per rank to form reduced shards; all-gather sends seven more to distribute them. Collective Communication Primitives explains the different output layouts.
At an assumed usable one-direction rate of 50 GB/s, the bandwidth term is 7/50 = 0.14 s. Full-duplex sending and receiving can happen concurrently; dividing 14 GB by the send rate would count both directions as serialized. Conversely, a vendor's bidirectional aggregate cannot be used as a one-direction send rate. Protocol overhead, startup, contention and late ranks remain outside this simple estimate.
Increasing to 64 ranks gives 2 × 63/64 × 4 = 7.875 GB sent per rank. The byte factor approaches two rather than growing linearly with rank count. That does not make scaling free: the topology, startup rounds and per-rank compute change too.
Check memory before believing the example. One possible FP32 Adam layout stores 4 bytes each for parameters, gradients and two moments: 16 GB of static tensor payload for this model. Activations, temporary buffers and allocator usage are extra. Other recipes use different formats and copies. An eight-GPU DDP group still needs a complete replica on each rank; eight memory capacities do not become one shared pool.
Then check overlap. DDP can start reducing a ready bucket while backward computes other gradients. It cannot promise that all but the final bucket finish before backward. Buckets may queue, or one rank may arrive late.
For a hypothetical timing trace, backward computation ends at 200 ms and the final required reduction completes at 240 ms. The exposed tail is 40 ms. This measures the end of backward computation, not the return of the DDP backward call, which must wait for synchronization. Summing every collective duration would overstate the delay if some ran alongside compute. Using the 140 ms bandwidth estimate as a guaranteed 140 ms addition would make the same mistake. Inspect the actual dependency path with torch.profiler and device activity timelines.
Start with DDP when a full replica fits and its measured step time meets the workload's needs. If memory prevents the required micro-batch, compare sharding or activation checkpointing. If memory is fine but communication dominates, examine rank placement, bucket scheduling and accumulation before changing the model layout.
What interviewers probe next
- What if ranks have unequal token counts? Average-of-rank-means can be wrong. Two tokens with mean gradient 1 and six with mean gradient 3 have global mean 2.5, not 2. Normalize against the actual global loss denominator.
- What does gradient accumulation save? At fixed effective batch, deferring synchronization for the first k−1 micro-batches reduces k sets of bucket reductions to one. Both forward and backward must be inside
no_sync()for those passes. Compared with one larger batch, this restores roughly the same gradient communication count; it does not cut it by another factor of k. - What if one rank sees nonfinite gradients? Check before applying updates and coordinate the step/skip decision across replicas. A rank must not independently skip a collective that peers will enter. Mixed-precision scaling and recovery must also remain consistent.
- What if the gradient dtype changes to BF16? For the same parameter count and ring model, the payload becomes 2 GB and the send count 3.5 GB. Numerical behavior and any communication hook need separate validation.
Common mistakes
- Treating "gradients dominate steady-state communication" as "parameters never move."
- Counting one all-reduce call when DDP actually issues multiple bucket reductions.
- Assuming equal rank means imply a token-weighted global mean.
- Adding all collective durations to a step even when they overlap with computation.
Key takeaways
- A 4 GB gradient buffer sends 7 GB per rank in an eight-rank ring, with another 7 GB received concurrently in the ideal full-duplex model.
- DDP retains a full training replica per rank; count activations and temporary memory too.
- Backward computation ending at 200 ms with reduction completion at 240 ms has a 40 ms exposed tail, regardless of the sum of kernel durations.