Cluster Bring-Up: Firmware, Drivers and the Stack

TL;DR: Bring-up has one correct order and every layer depends on the ones below it. Firmware (BMC, BIOS, CPLD, GPU VBIOS, NVSwitch, NIC) has to be at matched versions across the fleet before anything else, because a version skew produces intermittent faults that look like hardware. Then the operating system and a pinned kernel. Then the GPU driver, and on Blackwell parts that means the open kernel modules. Then the fabric manager, which is what makes an NVSwitch system present its GPUs as one NVLink domain, and on DGX and HGX B200 and B300 systems NVIDIA's documentation states that an OFED or MOFED driver plus libibumad3 and infiniband-diags are required because the NVSwitches are reached through a bridge device. Then the InfiniBand or Ethernet stack, then GPUDirect via the peer-memory module, then CUDA and NCCL, then the container runtime and the scheduler. Validate at each layer rather than at the end, because a failure at the top is unattributable.

The dependency chain

The dashed arrow is the reason the order matters. A firmware mismatch does not fail at step 1; it fails as an occasional NCCL hang or an XID during a large collective, six steps later, where nobody is looking at firmware.

What each step is, and what proves it worked

Two of these are the ones that go wrong most. The fabric manager version must match the driver version exactly; a mismatch leaves NVSwitch systems presenting isolated GPUs, and the symptom is a tensor-parallel job that works and is slow rather than one that fails. And the peer memory module is the difference between GPUDirect RDMA and a copy through host memory, which roughly halves fabric bandwidth. GPUDirect RDMA and GPUDirect Storage covers how to tell.

Validate at each layer, with the cost of not doing it

the cheap checks, in order, each one gating the next
  1. firmware inventory diff across the fleet          minutes, catches skew
  2. nvidia-smi on every node                          seconds per node
  3. ibstat: port state and rate on every port         catches the miscabled rail
  4. nvidia-smi topo -m: NVLink present between GPUs   catches a dead fabric manager
  5. nccl-tests all-reduce, single node, 8 GPUs        expect near the NVLink bus bandwidth
  6. nccl-tests all-reduce, 2 nodes                    catches GPUDirect being off, which
                                                       roughly halves the number
  7. nccl-tests all-reduce, full scalable unit         catches fabric and routing problems

what skipping them costs
  a fault found at step 7 on 512 GPUs has 512 nodes' worth of candidate causes
  the same fault found at step 3 has one
sanity: the whole ladder is a few hours of a technician's time on a cluster that costs
        512 x 24 x 2.5 = $30,720 per day to sit idle, so the ladder pays for itself the first
        time it catches anything

The version pinning discipline

Everything above is a version, and versions drift. The practices that hold up:

• One golden image, built once, with driver, OFED, fabric manager, CUDA and NCCL versions fixed inside it, and node provisioning that reimages rather than upgrades in place.
• Unattended upgrades off. A kernel that updates itself breaks the GPU driver at the next reboot, on one node, at an unpredictable time.
• The driver and fabric manager pinned together, since their versions must match.
• A change record for firmware, because a firmware rollout is the single most likely cause of a fleet-wide performance change and the investigation starts by asking what changed.

Containers, Images and GPU Cold Starts covers what runs above this line; the bring-up discipline is what makes the layer below the container reliable enough that a container problem is actually a container problem.

What interviewers are listening for

The order, and one dependency that is not obvious. Reciting "install the driver and CUDA" is what everyone says. Naming the fabric manager, saying it must match the driver version, and knowing that on Blackwell HGX systems it needs the InfiniBand stack present because the NVSwitches are reached through a bridge device, is the answer of someone who has done it. The second signal is validating per layer with nccl-tests at increasing scale, because that ladder is how bring-up problems stay attributable.

Key takeaways

• The order is firmware, OS and kernel, GPU driver, InfiniBand stack, fabric manager, peer memory, CUDA and NCCL, containers, telemetry.
• NVIDIA's documentation states that DGX and HGX B200 and B300 systems need an OFED or MOFED driver plus libibumad3 and infiniband-diags for the fabric manager to reach the NVSwitches.
• The fabric manager version must match the driver version; a mismatch leaves NVLink absent and the job merely slow rather than failed.
• Validate with nccl-tests at one node, two nodes, then the full unit, because a fault found at full scale has hundreds of candidate causes.
• Pin everything in a golden image, disable unattended upgrades, and keep a firmware change record.