Core

RDMA, InfiniBand and RoCEv2

Training across nodes moves hundreds of gigabytes per step, and a CPU-driven TCP stack cannot feed a 400 Gb/s link. RDMA lets a NIC write straight into a remote GPU's memory with no kernel and no copies, and it runs over two fabrics: InfiniBand, which is lossless by design, and RoCEv2, which is Ethernet made lossless by configuration. The choice is operational as much as technical, and the numbers that decide it are per-GPU bandwidth, the collective's volume, and who will debug a pause storm at 3 a.m.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

NCCL and Collective Algorithms Rail-Optimized and Fat-Tree Fabrics Congestion Control for AI Fabrics GPUDirect RDMA and GPUDirect Storage NVLink, NVSwitch and PCIe Debugging a Slow All-Reduce

LESSONS THAT TEACH THIS

Fleet and ReliabilityA GPU cluster has four networks and confusing them costs you a week14 min→

PRACTICE THIS IN REAL QUESTIONS

Networking, Interconnects & StorageInfiniBand or RoCE version 2 for a new GPU training cluster. Make the call and say what would reverse it.→GPU & Accelerator ArchitectureExplain GPUDirect RDMA. What does it remove from the data path, and when does it change performance?→Hardware, Cabling & Cluster Build-OutA vendor offers 15 percent off if you take their Ethernet fabric instead of InfiniBand. Decide.→Networking, Interconnects & StorageWhat is incast, why does it break an RDMA fabric, and what do ECN and priority flow control each do about it?→Networking, Interconnects & StorageWhat is RDMA, and why can a training cluster not just use TCP at 400 gigabits per second?→Hardware, Cabling & Cluster Build-OutAfter a maintenance window, two-node bandwidth halved. Single-node is unchanged. Diagnose it.→