What does nvidia-fabricmanager do, and what exactly breaks when it is not running?

It configures the NVSwitch fabric so eight GPUs present as one NVLink domain, and when it is absent nothing errors. Jobs run, results are correct, and everything multi-GPU is slower by a factor the team spends weeks not finding. The check that takes one command.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🖧 Hardware & Cluster Build-Out

Cluster Bring-Up: Firmware, Drivers and the StackBring-up is an ordered dependency chain and skipping a step produces a symptom that points somewhere else. Firmware first, then the operating system and kernel, then the GPU driver, then the fabric manager, then the network stack, then GPUDirect, then CUDA and NCCL, then the container and scheduling layer. On Blackwell HGX systems the fabric manager reaches the NVSwitches through a bridge device and therefore depends on the InfiniBand stack being present, which is a dependency that surprises almost everyone the first time.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

NVLink and Fabric FaultsThe links between GPUs are the part of a training node with the most connectors, the highest signalling rates and the least forgiveness: one marginal NVLink cable or one NVSwitch port turns an eight-GPU node into a straggler that slows a thousand-GPU job, and the symptom arrives as an NCCL timeout three layers away from the cause. This page covers what the links are, what their error counters mean, how a fault shows up in NCCL and in step time, how to isolate it to a GPU, a cable or a switch, and the arithmetic of why one degraded link is a whole-job problem.

Foundational

🖧 Hardware & Cluster Build-Out

NVLink Domains and the NVL72 RackAn NVLink domain is the set of GPUs that can address each other's memory at full fabric speed, and its size is the single most consequential number in a cluster design. Eight on an HGX node, 72 on a GB300 NVL72 rack. Inside the domain a collective moves at terabytes per second over a copper backplane; outside it, the same collective drops to the scale-out fabric at 800 Gb/s per GPU, a gap of roughly twenty times that decides how models are sharded.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

NVLink, NVSwitch and PCIeInside a node, GPUs talk over NVLink at 900 GB/s per H100 through an NVSwitch fabric that gives all eight cards full bandwidth to each other; to the host and to anything outside the node they talk over PCIe at 64 GB/s or a 400 Gb/s NIC at 50 GB/s. That fifteen-fold gap is why tensor parallelism stays inside the eight-GPU domain, why NVL72 changes the serving math for MoE, and why the question "how many GPUs share an NVLink domain?" is the first thing to ask about any cluster.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the physical layer for a multi-tenant GPU cloud. What changes versus a single-tenant cluster?Next in this trackWalk me through cabling a rail-optimized cluster. What goes where, and what goes wrong?Next in this trackThe cluster passed every acceptance test and production throughput is 20 percent below plan. Where do you look?

FEDITOR'S NOTE

Scored on knowing it configures the NVSwitch routing rather than being a driver, on the silent-failure characteristic, and on the one-command check plus the version-matching requirement.

DISCUSSION · 0

No comments yet — be the first to share your approach.