TL;DR: RDMA lets a network adapter access registered memory through supported operations with less host involvement in the data path. GPUDirect RDMA can extend that access to GPU memory, avoiding host staging. TCP can support training too; the choice depends on measured latency, CPU cost, topology and operational requirements.
How to approach it
Describe the memory path before quoting performance. Separate registration and setup from repeated transfers. Distinguish host-memory RDMA from direct GPU-memory access, and end with a matched measurement rather than a universal TCP throughput-per-core claim.
A strong answer
RDMA stands for remote direct memory access: a network adapter moves data straight into or out of registered memory on another machine, with the CPUs mostly out of the data path. A 400 Gb/s label tells you a nominal link rate, not how much training traffic reaches GPU memory. The payload might pass through host staging, share a PCIe path, or wait for another rank. RDMA reduces particular sources of overhead; it does not remove all of them.
A registered memory region gives the adapter an authorized mapping for the relevant buffer. Registration and lifetime management involve the OS and driver; established transfers can use a path with less kernel involvement. Reuse registration where appropriate, but never keep a stale mapping after the allocation is released or repurposed.
Applications submit work requests through queues. In a common reliable-connected verbs design, a queue pair has send and receive work queues, and completion queues report completed work. A two-sided send requires matching receive resources. A one-sided read or write accesses an authorized remote region without the remote application posting a receive for each payload. Setup, permissions, synchronization and notification still require a protocol between participants.
Kernel bypass and avoiding copies are related but distinct. Kernel bypass reduces operating-system work during repeated transfers; avoiding copies removes movement between intermediate buffers. TCP implementations can use batching, offloads and copy-avoidance techniques, so there is no universal five-gigabit-per-core ceiling or fixed fifty-core requirement for a 400 Gb/s link.
Here is a synthetic CPU-budget comparison, not a benchmark:
400 Gb/s / 8 = 50 GB/s nominal, before overhead
at 40 GB/s application payload and measured cost 2 CPU cycles/byte:
CPU demand = 80 billion cycles/s
at an assumed available 3 billion cycles/s per core: about 26.7 core-equivalents
if measured cost is 0.2 cycles/byte instead: about 2.67 core-equivalents
sanity: the measured cycles/byte changes the answer tenfold.
Core-equivalents are an accounting estimate, not a promise of perfect scaling. NUMA placement, polling, contention and CPU frequency affect the actual result. Measure application payload and CPU time together rather than inferring CPU cost from the link label.
Host-memory RDMA does not by itself eliminate GPU staging. NVIDIA's GPUDirect RDMA documentation describes direct device-memory access and its platform and memory-ordering requirements. Support depends on the driver, memory registration mechanism and PCIe topology. A direct path can avoid host-buffer traffic, but it does not guarantee exactly twice the bandwidth: different links may overlap or become the next bottleneck.
Completion and visibility matter as much as speed. Do not assume that a concurrently running GPU consumer can safely read data merely because a remote write was issued. Use the supported synchronization and library contract, and keep buffers alive until the relevant work completes. Correctness tests should exercise reuse and overlapping transfers, not only a single isolated copy.
Compare a matched host-memory transport test, a supported GPU-memory test, representative collectives and the full training step. A good two-node result does not certify every rail, rank, switch path or congested workload. GPUDirect RDMA and GPUDirect Storage links the memory paths to deployment concerns.
Use RDMA where its measured savings justify the supported stack and operations burden. Keep ordinary sockets when they meet the workload's latency and CPU budgets. Reverse a direct-path preference if the topology is unsupported or the end-to-end workload does not improve; preserve correctness and supportability first.
What interviewers probe next
- "Does one-sided mean no remote CPU ever?" No; setup and application coordination remain, even if each payload avoids remote CPU handling.
- "Is all RDMA InfiniBand?" No. RDMA can use other transports, including RoCE; fabric configuration still matters.
- "Does a TCP test tell me nothing?" It measures that TCP path. It just does not establish RDMA or GPU-buffer performance.
Common mistakes
- Assuming hardware capability proves the selected path is active.
- Treating reduced host staging as a universal 2× gain.
- Reusing a buffer before transfer completion and visibility are established.
Key takeaways
- Registration, access permission and buffer lifetime enable direct memory access safely.
- 400 Gb/s is 50 GB/s nominal; useful payload and CPU cost require measurement.
- Choose the path using complete-workload evidence, not a fixed TCP-versus-RDMA slogan.