rdma

Without it, every byte a GPU sends over the network crosses PCIe twice and passes through host memory. With it, the NIC reads and writes GPU memory directly. The PCIe arithmetic that shows why a 400G NIC cannot be fed the slow way, the NCCL line that tells you which path you are on, and the checkpoint case.

Single-node fine and two-node halved is a specific signature, and it points at one thing before anything else. Why the copy path costs exactly about half, the five causes that produce it, and the one log line that confirms which.

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

Both carry RDMA at the same line rate. One arrives lossless because the transport was designed that way, the other becomes lossless only if a set of switch settings is correct on every port. What that difference costs in operations, what it saves in money and hiring, and the fleet size where the answer flips.

A single core moves a few gigabits per second of TCP once copies and interrupts are counted, so filling one 400 gigabit link would take most of a server's cores doing nothing but networking. What RDMA removes, what a queue pair actually is, and where the GPU fits when the data never belongs to the host at all.

A pod's default network is an overlay that no RDMA NIC will speak. The three ways to give a pod the fabric (host networking, SR-IOV virtual functions via a NIC device plugin, or a macvlan secondary interface), what each costs in isolation and speed, how NCCL finds the right NIC, and the failure that halves bandwidth.

Prefill is compute-bound and decode is bandwidth-bound, and one GPU running both does each worse. The pool ratio from the traffic's prompt and output lengths, the KV transfer cost per request and the link it needs, the router that pairs a prefill worker with a decode worker, and the load at which it beats a monolith.