How do you give a Kubernetes pod RDMA access to the fabric for NCCL, and what goes wrong when it is done naively?

A fabric address, an RDMA device and a working GPU-memory registration path are separate requirements. Verify each inside the pod, then test the actual GPU-to-NIC placement and collective behavior.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

A fabric address, an RDMA device and a working GPU-memory registration path are separate requirements. Verify each inside the pod, then test the actual GPU-to-NIC placement and collective behavior.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🗂️ Scheduling & Orchestration

Kubernetes GPU SchedulingKubernetes coordinates GPU discovery, scheduling and container access through several components. Understanding their separate roles makes allocation failures easier to diagnose.

Core

🔌 Networking & StorageSign in

RDMA, InfiniBand and RoCEv2RDMA lets network adapters transfer data directly between registered memory regions. InfiniBand and RoCE provide different network and operating environments for those transfers.

Advanced

🔌 Networking & Storage🔒 Premium

GPUDirect RDMA and GPUDirect StorageGPUDirect enables supported devices to transfer data to or from GPU memory without a host staging buffer. Its benefit depends on topology, software support and the transfer path.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

GPU Tenant Isolation BoundariesGPU allocation, virtual machines and network policies protect different boundaries. Trace each tenant's device, network and storage paths, then test both permitted work and forbidden access before declaring a shared GPU service isolated.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackUpgrade the GPU driver across 2,000 live nodes without breaking running jobs. Walk me through the plan and what can go wrong.Next in this trackA 64-node gang-scheduled job has been pending for six hours while the cluster shows free GPUs. Debug it live.Closest to this topicExplain GPUDirect RDMA. What does it remove from the data path, and when does it change performance?

FEDITOR'S NOTE

Distinguish network attachment from device access and GPUDirect. Do not infer a unique fault from bandwidth or compare aggregate NIC line rate directly with a per-rank collective metric.

DISCUSSION · 0

No comments yet — be the first to share your approach.