Advanced

Ring vs Tree All-Reduce

A ring all-reduce moves the minimum possible bytes per rank but takes 2(N-1) steps, so its latency grows with the number of GPUs; a tree finishes in a logarithmic number of steps but is harder to keep bandwidth-optimal. NCCL keeps both, chooses per message size and rank count, and reading its choice is how you diagnose a collective that is slower than the fabric allows.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Collective Communication Primitives Data Parallelism and DDP NCCL and Collective Algorithms Debugging a Slow All-Reduce NVLink, NVSwitch and PCIe Topology-Aware Communication

LESSONS THAT TEACH THIS

Training at ScaleMap the mesh so the chattiest dimension takes the shortest path14 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismDerive the cost of a ring all-reduce. Why is it bandwidth-optimal, and where does it stop scaling?→Napkin Math, Cost & CapacityHow many tokens per second can a 70B model generate for a single user on H100s?→Napkin Math, Cost & CapacityEstimate the latency of one decode step for a 70B model under tensor parallelism across eight H100s→Coding for InfraSimulate a ring all-reduce on arrays. Verify both the result and the bytes each rank sends.→Networking, Interconnects & StorageHow would you know GPUDirect RDMA is not being used, given that nothing reports an error when it is disabled?→Hardware, Cabling & Cluster Build-OutAfter a maintenance window, two-node bandwidth halved. Single-node is unchanged. Diagnose it.→