Advanced

Ring vs Tree All-Reduce

Trace one chunk through a ring, separate byte balance from startup rounds, and compare tree depth without assuming identical bandwidth. Use measured message sizes and topology to choose.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Collective Communication Primitives Data Parallelism and DDP NCCL and Collective Algorithms Debugging a Slow All-Reduce NVLink, NVSwitch and PCIe Topology-Aware Communication

LESSONS THAT TEACH THIS

Training at ScaleMap communication groups onto measured physical paths24 min→Training at ScalePrice parallelism by message size, frequency and exposed time26 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismDerive the cost of a ring all-reduce. Why is it bandwidth-optimal, and where does it stop scaling?→Coding for InfraSimulate a ring all-reduce on arrays. Verify both the result and the bytes each rank sends.→Napkin Math, Cost & CapacityHow many tokens per second can a 70B model generate for a single user on H100s?→Napkin Math, Cost & CapacityEstimate the latency of one decode step for a 70B model under tensor parallelism across eight H100s→Networking, Interconnects & StorageHow would you know GPUDirect RDMA is not being used, given that nothing reports an error when it is disabled?→Hardware, Cabling & Cluster Build-OutAfter a maintenance window, two-node bandwidth halved. Single-node is unchanged. Diagnose it.→