How much network bandwidth does data-parallel training need?

TL;DR: Each DDP step all-reduces the full gradient, 2 bytes per parameter in bf16. A ring all-reduce moves 2(n−1)/n × that per rank, so a 7B model sends and receives about 24.5 GB per rank per step on 8 ranks; at a 1 s step that is 245 Gb/s per GPU, which NVLink covers with room and a 400 Gb/s NIC covers only with overlap. For a 70B the volume is 247 GB per rank per step and the step must be long, or the parallelism must change.

How to approach it

Ask the model size, the precision of the gradients (bf16 is standard), the number of ranks and the step time, and whether the ranks share NVLink or talk over the inter-node fabric. Compute gradient bytes first, then apply the ring factor, then divide by step time to get a bandwidth requirement with a unit. Compare it to the link the ranks actually use, and then say the sentence about overlap: the requirement is hidden while the reduction finishes before backward does.

A strong answer

A typical situation: a cluster is specified with a fabric chosen from a vendor's recommendation rather than from the gradient bytes it has to carry per step. The requirement is two numbers divided by the step time.

Data parallelism gives each rank a copy of the model and a slice of the batch; after backward, the ranks average their gradients so every copy applies the same update. The all-reduce over n ranks with a ring algorithm has each rank send and receive 2(n−1)/n times the buffer: a reduce-scatter pass and an all-gather pass, each moving (n−1)/n of the buffer.

inputs:  N = 7e9 parameters, gradients in bf16 = 2 B each
         n = 8 ranks (one node), step time = 1.0 s

gradient bytes = N × 2 = 7e9 × 2 = 14 GB

per-rank traffic per step (ring) = 2 × (n−1)/n × gradient bytes
                                 = 2 × 7/8 × 14 GB
                                 = 24.5 GB sent and 24.5 GB received

bandwidth needed = 24.5 GB ÷ 1.0 s = 24.5 GB/s = 196 Gb/s per rank each direction

sanity: NVLink on an H100 is 900 GB/s bidirectional, about 450 each way, so the 8-rank
        intra-node case takes 24.5 ÷ 450 ≈ 55 ms of link time per step, 5% of the step.
        A 400 Gb/s NIC (50 GB/s) would take 0.49 s, half the step, which is only acceptable
        if it fully overlaps with backward.

The same chain for a 70B shows why DDP alone stops being the right parallelism:

N = 70.6e9, bf16 gradients = 141 GB
per rank per step (8 ranks) = 2 × 7/8 × 141 = 247 GB
over NVLink at 450 GB/s each way: 0.55 s
over a 400 Gb/s NIC (50 GB/s):    4.9 s

A 70B step on 8 GPUs is a few seconds at most, so a 4.9 s reduction over the NIC would dominate; even the NVLink time is a large fraction. That is one of the reasons 70B training uses tensor parallel inside the node and data parallel across nodes: the cross-node all-reduce then carries each TP group's gradient shard (141 ÷ 8 ≈ 17.6 GB per rank per step), and ZeRO or FSDP sharding replaces the all-reduce with a reduce-scatter plus an all-gather, which moves the same bytes but in pieces that are easier to overlap. The Communication Volume Estimates page carries the per-scheme table.

Overlap is the term that decides whether the requirement is visible. DDP reduces gradients in buckets as backward produces them, so the reduction for layer L runs while backward computes layer L−1. The bandwidth requirement is then "reduction time ≤ backward time", roughly two thirds of the step. When that holds, the network is invisible; when it fails, every step has an exposed communication tail and MFU drops in proportion. A useful diagnostic is the step-time gap: if the step is much longer than forward plus backward measured on one rank, the difference is exposed communication.

overlap budget for the 7B on the 400 Gb/s NIC:
  backward ≈ 2/3 × 1.0 s = 0.67 s available
  reduction = 0.49 s → fits, with 0.18 s of margin
  if the step shrinks to 0.5 s (bigger fleet, same global batch): reduction 0.49 s > 0.33 s → exposed

So the requirement scales with 1 ÷ step time, and step time shrinks as the fleet grows at fixed global batch. Beyond a few hundred ranks the ring's (n−1)/n factor is flat but the latency of the collective grows, and tree and hierarchical algorithms take over.

The reversal condition, and the decision: intra-node DDP on NVLink needs no thought at 7B; cross-node DDP needs a 400 Gb/s class fabric with RDMA and gradient bucketing, and a step long enough (about 1 s or more for a 7B) to hide the reduction; at 70B, switch the intra-node dimension to tensor parallel and shard the optimizer. The reversal is gradient compression or lower-precision reduction (fp8 gradients, or reduced-precision all-reduce), which halves the bytes at a numerics risk. Communication Volume Estimates carries the same arithmetic across every other parallelism axis. Data Parallelism and DDP is the mechanism these bytes belong to.

What interviewers probe next

• "Where does 2(n−1)/n come from?" A ring reduce-scatter moves (n−1)/n of the buffer through each rank, then an all-gather moves the same again; the total approaches 2x the buffer as n grows.
• "What if I use FSDP?" Same total bytes per step per rank (a reduce-scatter of gradients plus an all-gather of parameters, both (n−1)/n), but the parameter all-gather sits in the forward pass and adds a second overlap window to manage.
• "How do I know the network is the bottleneck?" Compare step time to forward plus backward on one rank; watch NCCL's per-collective time with NCCL_DEBUG=INFO timing or a profiler trace showing the all-reduce extending past the end of backward.
• "Does a larger global batch help?" Yes: the gradient bytes per step are fixed, so doubling the per-rank batch doubles the step time and halves the bandwidth requirement.

Common mistakes

• Using 4 bytes per parameter (fp32) for the gradient; in mixed precision the reduced gradient is bf16, 2 bytes.
• Forgetting the ring factor and reporting 14 GB when the wire carries 24.5.
• Quoting the NIC in Gb/s and the volume in GB without converting; the factor of 8 is the whole answer.
• Treating the requirement as a hard bandwidth number rather than a bandwidth over the overlap window.

Key takeaways

• DDP all-reduce volume per rank per step ≈ 2(n−1)/n × N × 2 B: 24.5 GB for a 7B on 8 ranks, 247 GB for a 70B.
• Divide by step time for a bandwidth requirement; a 7B at 1 s needs about 200 Gb/s per rank each way.
• Overlap hides it while reduction time ≤ backward time (about two thirds of the step).
• At 70B, DDP on the NIC does not fit; use TP inside the node and shard the optimizer across nodes.