TL;DR: Specify the gradient dtype and collective group, then count bytes over the available communication window. For an illustrative 7B model with BF16 gradients and an eight-rank ring, each rank sends 24.5 GB and receives 24.5 GB. Over one second that is 196 Gb/s in each direction. Bucket readiness and the actual path determine whether communication is hidden.
How to approach it
Start with a feasible training layout and the bytes actually reduced. Distinguish ranks from nodes and sent traffic from bidirectional totals. Calculate a bandwidth term, then inspect overlap rather than assuming the whole backward interval is available to every byte.
A strong answer
A fabric recommendation says “400G per GPU,” but the training recipe has changed its gradient format and local batch. Recalculate the workload before judging the recommendation. In data parallelism (DDP, PyTorch's DistributedDataParallel), every GPU holds a full model copy, trains on different data, and averages gradients with its peers after each backward pass. Data Parallelism and DDP explains gradient synchronization; Communication Volume Estimates covers the byte accounting across layouts.
Assume 7 billion parameters with two-byte communicated gradients and eight ranks in one ring group. This is an explicit recipe assumption: mixed-precision forward execution does not guarantee BF16 gradients. FP32 gradients double these bytes. Check that the rest of the training state and activations fit each rank too.
A ring all-reduce passes chunks around the ranks in two phases: one adds the gradients together, one shares the finished sums. In each phase a rank sends (N − 1)/N of the payload, so it sends 2 × (N − 1)/N in total.
gradient payload = 7e9 × 2 = 14e9 bytes
ring factor = 2×(8−1)/8 = 1.75
sent per rank = 14×1.75 = 24.5 GB
received per rank = 24.5 GB
one-second average send demand = 24.5 GB/s = 196 Gb/s
sanity: 24.5×8 = 196; Gb/s and GB/s differ by eight.
NCCL tests' performance definitions explain the normalization. A real collective may use a different algorithm or hierarchy; this ring model is not a direct NIC counter. Traffic can share links or remain inside a node. Full-duplex directions can overlap, so dividing sent-plus-received bytes by a one-direction rate would overcharge time.
A nominal 400 Gb/s link has 50 GB/s raw capacity before overhead. At an assumed achieved payload rate of 40 GB/s, the 24.5 GB serialization component is 612.5 ms. Do not use 's aggregate bidirectional NVLink specification as one-direction ring bandwidth or assume a particular topology realizes it.
The overlap question needs a timeline. A gradient bucket becomes eligible only after its required local gradients are ready, and completion may wait for other ranks. A total 700 ms backward phase does not automatically hide a 612.5 ms reduction. If the final bucket is ready at 680 ms and needs 50 ms after that, its completion is at least 730 ms: a 30 ms tail after compute ends at 700 ms, under this synthetic serial bucket example.
| Change | Recompute or measure |
|---|---|
| BF16 to FP32 gradients | Twice the payload at unchanged parameter count |
| More ranks at fixed global batch | Less compute per rank may reduce overlap time |
| Accumulation with deferred sync | Reduction frequency per optimizer update |
| TP or PP added | Which parameter shard each DP group reduces |
| FSDP/ZeRO added | Parameter gathers, gradient reductions and lifetimes |
For 70.6B parameters under the same BF16 ring assumptions, the gradient payload is 141.2 GB and per-rank send volume is 247.1 GB. This traffic calculation does not establish a feasible plain-DDP training replica on an 80 GB GPU. Sharding or another supported layout changes memory ownership and communication, so calculate the actual groups before sizing a fabric.
Use a profiler to compare collective completion with compute dependencies. NCCL_DEBUG=INFO can help identify transports and configuration, but it is not a complete timing instrument for exposed delay. Compare representative messages and rank placement with nccl-tests, then restore application load. A faster isolated collective may still lose overlap in the training schedule.
I would choose the layout and network from measured step time at the required global batch and reliability envelope. Reverse a communication compression experiment if update quality or conversion overhead outweighs the byte saving. There is no universal step-duration or model-size threshold at which a 400G link becomes sufficient.
What interviewers probe next
- “Does FSDP move the same bytes?” Not generally. Gather frequency, resharding policy, parameter and gradient dtypes, and accumulation change the accounting.
- “Does doubling batch halve required bandwidth?” Gradient bytes per update may stay fixed, but step time need not double and the training objective changes unless the effective batch is controlled.
- “What is the key overlap signal?” The required collective's completion on the update's critical path, not the sum of all collective durations.
Common mistakes
- Declaring BF16 gradients from BF16 autocast alone.
- Charging both full-duplex directions as serialized traffic.
- Treating backward duration as a guaranteed overlap window for every bucket.
Key takeaways
- The stated 7B ring sends 24.5 GB per rank, or 196 Gb/s over one second.
- At an assumed 40 GB/s, its bandwidth component is 612.5 ms.
- Actual gradient dtype, rank mapping and readiness determine the network decision.