TL;DR: TCP costs the host CPU: a system call per operation, a copy from user memory into kernel buffers, interrupt handling, and the protocol work itself. A single core sustains a few gigabits per second of that, so saturating one 400 Gb/s link would take on the order of forty to eighty cores doing nothing else, which is most of a server bought to feed GPUs. RDMA removes both costs. The application registers a memory region once, then posts work requests to a queue pair that the network adapter reads directly, so the kernel is not on the data path and the adapter reads and writes application memory without a copy. What remains is a doorbell write and a completion, a few microseconds of latency rather than tens. GPUDirect RDMA extends the same idea to the accelerator: the adapter reads from and writes to GPU memory across PCIe without staging through host memory, which removes a bounce that would otherwise halve effective bandwidth and put the host back in the path.
How to approach it
Start with the cost of the alternative, because the case for RDMA is arithmetic rather than preference. Separate the two things it removes, kernel involvement and copies, since they are often merged and they save different amounts. Then describe the mechanism concretely enough to be credible: registered memory, queue pairs, work requests, completions. Close with the GPU, which is where the remaining bounce lives and where most of the practical bugs are.
A strong answer
A typical situation: a team benchmarks their fabric with a TCP tool, sees 25 Gb/s between two nodes, and concludes the network is underperforming. The network is fine. They measured how much TCP one core can push, and the fix is not tuning the network stack.
The arithmetic that makes the case:
TCP cost per byte on the host: a copy from user to kernel buffers, interrupt or poll handling,
checksum and protocol work, and a context switch per system call
sustained TCP throughput per core, with a tuned stack and large frames: roughly 5 to 10 Gb/s
one 400 Gb/s link needs 400 / 8 = 50 cores at the optimistic end,
400 / 5 = 80 at the realistic end
a training node has 8 GPUs and roughly 100 to 200 CPU cores, and those cores are supposed to
be running the data loader, the framework and the job
so filling one NIC with TCP would consume most of the
host, and a node has 8 NICs
RDMA per-core cost: a doorbell write to post a work request, and a completion queue poll.
The adapter does the transfer. One core can drive several hundred gigabits per second
sanity: the ratio here is roughly two orders of magnitude in CPU per byte, which is why this
is not a tuning question. No amount of TCP tuning closes a 50-core gap
What RDMA actually removes, as two separate savings:
kernel bypass
the application talks to the adapter through a mapped queue rather than through system calls
removes: system call overhead, context switches, the kernel's protocol processing
gains mostly latency and CPU: a message goes out in a few microseconds instead of tens
zero copy
the adapter reads directly from the application's registered memory
removes: the copy from user memory into kernel socket buffers, and back on the far side
gains mostly bandwidth and CPU: at 400 Gb/s a single memory copy of every byte is 50 GB/s
of memory bandwidth spent on nothing
these are independent. A design could bypass the kernel and still copy, or copy less without
bypassing, and both halves are needed to reach line rate on one core
The mechanism, concretely:
memory region the application registers a buffer with the adapter once. The kernel pins the
pages so they cannot be swapped or moved, and the adapter records the address
translation. Registration is expensive; it happens at startup, not per message
queue pair a send queue and a receive queue that the application writes to directly.
The pair is the connection: two queue pairs on two nodes are bound to each
other, and the fabric routes between them
work request an entry the application posts describing an operation: send this region,
or write these bytes into that remote address
completion queue the adapter posts an entry when an operation finishes; the application polls
or waits on it
one-sided ops an RDMA write lands data in the remote node's memory with no involvement from
the remote CPU at all, which is the property that makes collectives efficient
The GPU is where the remaining bounce lives, and where the practical failures are. Without GPUDirect RDMA the data path is GPU memory to host memory over PCIe, then host memory to the adapter over PCIe, so every byte crosses the bus twice and the host allocates and manages the staging buffer:
without GPUDirect: GPU -> host (PCIe) -> NIC (PCIe), two crossings plus a host buffer
with GPUDirect: GPU -> NIC directly over PCIe, one crossing, no host memory involved
effect: roughly half the PCIe traffic and the removal of host memory bandwidth and a copy
from the path entirely
requirement: the NIC and the GPU should sit under the same PCIe switch for the direct path to
be worth taking, which is why nodes are built with a NIC per GPU on matching roots
GPUDirect RDMA and GPUDirect Storage covers the setup and the checks. The common failure is that it is silently off: the module is not loaded, or the NIC and GPU are on different PCIe roots so the library declines to use it, and the job runs at roughly half the bandwidth with no error. Verifying it is a line in the NCCL debug output rather than a guess.
The reversal condition: none of this is worth the complexity for traffic that is small or infrequent. Control-plane messages, metrics, logs and checkpoint uploads run perfectly well over TCP, and putting them on RDMA adds registration cost and a more fragile path for no benefit. The rule that holds: RDMA for the collectives that NCCL and Collective Algorithms describes and for any bulk path on the critical path of a step, ordinary sockets for everything else, which is also why a training node has both a fabric NIC and a conventional one.
What interviewers probe next
- "Why does memory have to be registered?" The adapter uses physical addresses and cannot tolerate the operating system moving or swapping the pages under it, so registration pins them and records the translation.
- "What is a one-sided operation?" A write or read that completes without the remote CPU participating. Collectives use them so a receiver does not need to be scheduled to make progress.
- "Does RDMA help small messages too?" Yes, and mostly through latency: a few microseconds against tens for a TCP round trip, which matters for the many small collectives a tensor-parallel layer issues.
- "What replaces TCP's congestion control?" On InfiniBand, credit-based flow control at the link layer. On RoCE, priority flow control plus ECN and a congestion algorithm, configured on the switches.
Common mistakes
- Benchmarking the fabric with a TCP tool and concluding the network is slow.
- Treating kernel bypass and zero copy as one thing, when they save different resources.
- Assuming GPUDirect is on because the hardware supports it, when a missing module or a mismatched PCIe root silently disables it.
- Moving every kind of traffic to RDMA, including control-plane messages that gain nothing and add fragility.
Key takeaways
- One core sustains roughly 5 to 10 Gb/s of TCP, so a 400 Gb/s link would need 50 to 80 cores; RDMA drives it with one.
- Two separate savings: kernel bypass removes system calls and protocol work, zero copy removes a copy of every byte.
- The mechanism is registered memory, queue pairs, posted work requests and completions, with one-sided writes that need no remote CPU.
- GPUDirect removes the host bounce, halving PCIe crossings; it fails silently when the module is missing or the NIC and GPU are on different roots.