nvlink

An H100 SXM moves 900 GB/s over NVLink; a PCIe card moves about 64 GB/s per direction over its slot. Whether that 7x to 14x gap costs you anything depends on how many bytes your parallelism strategy sends per step, and the answer is very different for tensor and data parallelism.

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

Half the expected bus bandwidth has four plausible causes needing different fixes: a link trained at reduced width, a fabric that never came up, a topology the job did not expect, or a test that measured latency. The sequence of commands that tells them apart, and what each output looks like on a healthy node.

The part with no NVLink can be the correct choice, and the reason is the workload shape rather than the specification. What fits on one card, what MIG partitioning buys a multi-tenant service, and the exact point where the missing scale-up link makes the decision flip.

It configures the NVSwitch fabric so eight GPUs present as one NVLink domain, and when it is absent nothing errors. Jobs run, results are correct, and everything multi-GPU is slower by a factor the team spends weeks not finding. The check that takes one command.

Three ways to split a training job, one table, and the rule that places each of them: activations on NVLink, gradients on the fabric, stage boundaries in between. With the byte counts that justify the placement.

Four all-reduces per transformer block, on activations, on the critical path. Inside the NVLink domain they cost 8% of a step; across the NIC they cost more than the step itself. The derivation for a 70B, and the second reason TP stops that has nothing to do with the network.

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

The ring all-reduce formula, 2(n−1)/n times the buffer over the link bandwidth, applied to 141 GB of bf16 gradients on an NVLink node: 0.27 s by the formula, and the two reasons the measured number is different.

Three parallelism dimensions, two link speeds eighteen times apart, and one mapping that makes the job fast. The traffic each dimension generates per step, why the loudest one must stay inside the node, and the arithmetic showing what a wrong rank order costs before anyone notices.

Two all-to-alls per layer across sixty layers is over a hundred collectives on the critical path of every decode step, each one small and latency-bound. The per-GPU volume worked out, the time on each link type, and why a single large NVLink domain changes the design rather than just improving it.

Same job, same GPU count, half the speed: the slow one was split 4 and 4 across two nodes and its tensor-parallel all-reduces run at fabric speed instead of NVLink speed. The bandwidth arithmetic, the per-parallelism placement rule, and the scheduler policy that enforces it.

A published run gives the numbers directly: 419 unexpected interruptions in 54 days on 16,384 GPUs, and about three quarters of them hardware. What that implies per GPU-hour, which components dominate, and why a fleet above a few thousand GPUs must treat failure as routine rather than exceptional.

The driver logs a numbered code when something goes wrong on a GPU, and the number tells you whether to retry the job, drain the node or file a hardware return. The codes worth memorizing in three groups, the action each implies, and the automation that turns a log line into a drained node.

A link that corrupts occasionally does not fail; it retransmits, and each retransmission costs bandwidth that shows up as a slow job with no error anywhere. The counters that reveal it, the arithmetic turning a replay rate into lost bandwidth, and the threshold where watching becomes draining.

Prefill is compute-bound and decode is bandwidth-bound, and one GPU running both does each worse. The pool ratio from the traffic's prompt and output lengths, the KV transfer cost per request and the link it needs, the router that pairs a prefill worker with a decode worker, and the load at which it beats a monolith.