How would you know GPUDirect RDMA is not being used, given that nothing reports an error when it is disabled?

It fails open: the data still moves, the job still runs, and everything is about half as fast. The one log line that settles it, the four reasons the library declines to use it, and why the bandwidth signature is close enough to exactly half to be diagnostic on its own.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

GPUDirect RDMA and GPUDirect StorageBy default a byte leaving a GPU for the network or the disk makes a detour through host memory, crossing PCIe twice and costing a CPU copy. GPUDirect RDMA lets the NIC read and write GPU memory directly, and GPUDirect Storage does the same for NVMe. The win is not raw bandwidth (PCIe is the ceiling either way) but the halving of PCIe traffic and the removal of the host as a bottleneck, which is what makes collectives run at NIC rate and checkpoints run at drive rate. When it is silently off, everything still works, at half speed.

Advanced

🕸️ Distributed Training🔒 Premium

Ring vs Tree All-ReduceA ring all-reduce moves the minimum possible bytes per rank but takes 2(N-1) steps, so its latency grows with the number of GPUs; a tree finishes in a logarithmic number of steps but is harder to keep bandwidth-optimal. NCCL keeps both, chooses per message size and rank count, and reading its choice is how you diagnose a collective that is slower than the fabric allows.

Core

🔌 Networking & StorageSign in

RDMA, InfiniBand and RoCEv2Training across nodes moves hundreds of gigabytes per step, and a CPU-driven TCP stack cannot feed a 400 Gb/s link. RDMA lets a NIC write straight into a remote GPU's memory with no kernel and no copies, and it runs over two fabrics: InfiniBand, which is lossless by design, and RoCEv2, which is Ethernet made lossless by configuration. The choice is operational as much as technical, and the numbers that decide it are per-GPU bandwidth, the collective's volume, and who will debug a pause storm at 3 a.m.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign storage for a multi-tenant GPU cloud. How do you keep one customer's checkpoint burst from slowing another's training run?Next in this trackAfter a switch firmware update, all-reduce bandwidth across the cluster fell 20 percent. Every link is up and no errors are logged. Find it.Closest to this topicAfter a maintenance window, two-node bandwidth halved. Single-node is unchanged. Diagnose it.

FEDITOR'S NOTE

Scored on knowing the failure is silent and the bandwidth signature is about half, on the specific log line and topology check, and on the four causes with a distinct fix each.

DISCUSSION · 0

No comments yet — be the first to share your approach.