AI Infra Interviews logo
GPU & Accelerator Architecture / 30
expertNewNVIDIACoreWeaveCrusoe

One node in your cluster shows half the expected NVLink bandwidth in nccl-tests. Walk me through isolating it.

Half the expected bus bandwidth has four plausible causes needing different fixes: a link trained at reduced width, a fabric that never came up, a topology the job did not expect, or a test that measured latency. The sequence of commands that tells them apart, and what each output looks like on a healthy node.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational
🔌 Networking & Storage
Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.
Advanced
🧩 GPU & Accelerator Architecture🔒 Premium
NVLink, NVSwitch and PCIeInside a node, GPUs talk over NVLink at 900 GB/s per H100 through an NVSwitch fabric that gives all eight cards full bandwidth to each other; to the host and to anything outside the node they talk over PCIe at 64 GB/s or a 400 Gb/s NIC at 50 GB/s. That fifteen-fold gap is why tensor parallelism stays inside the eight-GPU domain, why NVL72 changes the serving math for MoE, and why the question "how many GPUs share an NVLink domain?" is the first thing to ask about any cluster.
Foundational
🔌 Networking & Storage
NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.
Foundational
🩺 Fleet Reliability & Observability
GPU Failure Modes and XID ErrorsWhen a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.
UP NEXT ON YOUR JOURNEY
FEDITOR'S NOTE

Scored on a diagnostic order that goes from cheapest check to most invasive, correct expectations for each command's output on a healthy HGX node, and the ability to distinguish a hardware fault from a software mapping error from a measurement artifact.

DISCUSSION · 0

No comments yet — be the first to share your approach.