← Napkin Math, Cost & Capacity / 19

mediumNVIDIAMeta

How long does an all-reduce of a 70B model's gradients take on eight GPUs?

Derive ring all-reduce traffic, match directional bytes to bandwidth, and check memory feasibility before estimating the communication exposed on a training step.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Derive ring all-reduce traffic, match directional bytes to bandwidth, and check memory feasibility before estimating the communication exposed on a training step.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🕸️ Distributed Training

Data Parallelism and DDPReplicas combine gradients from different data. Follow a two-worker update, fix unequal-token weighting, and see why queued gradient buckets can leave more than one reduction exposed.

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL implements GPU collectives using algorithms, protocols and channels over available hardware paths. Learn what each choice changes, how to read bandwidth results, and which evidence separates a communication problem from late application work.

🔌 Networking & Storage

Debugging a Slow All-ReduceInterpret NCCL algbw and busbw, then isolate a slow collective with controlled tests. Distinguish rank delays, GPU-memory paths and shared-fabric limits without guessing from bandwidth ratios.

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits operations within a layer across GPUs. Derive the column/row matrix split, count training and inference collectives, and choose a group size from memory, latency and actual interconnect performance.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow many tokens per second can one B200 decode for a 70B model?Next in this trackWhat storage throughput does a 16k-GPU training cluster need?Next in this trackTraffic peaks at three times the daily average. Capacity-plan the serving fleet.

FEDITOR'S NOTE

Catch the bidirectional bandwidth error, distinguish all-reduce from sharded schedules, and reason about bucket readiness.

DISCUSSION · 0

No comments yet — be the first to share your approach.