New nodes arrive with no software. Walk me through bring-up, and say why the order matters.

TL;DR: Nine layers, in one order. Firmware first, at matched versions across the whole fleet: BMC, BIOS, CPLD, GPU VBIOS, NVSwitch and NIC. Then the operating system with a pinned kernel and unattended upgrades disabled. Then the GPU driver, which on Blackwell parts means the open kernel modules. Then the InfiniBand stack, because NVIDIA's documentation states that DGX and HGX B200 and B300 systems need an OFED or MOFED driver plus libibumad3 and infiniband-diags for the fabric manager to reach the NVSwitches through their bridge device. Then nvidia-fabricmanager, at a version matching the driver exactly. Then nvidia-peermem for GPUDirect RDMA. Then CUDA and NCCL. Then the container toolkit and the scheduler's device plugin. Then telemetry. Validate after each layer with a command that produces a number, and run nccl-tests at one node, two nodes and full scale in that order, because a fault found at full scale has hundreds of candidate causes and the same fault found at one node has one.

How to approach it

Give the order and the reason each layer depends on the one below. Name the check that proves each layer worked, because "installed" is not "working". Then the validation ladder at increasing scale. Then the pinning discipline, since everything above is a version and versions drift. Close with the failure that motivates the whole thing, which is a firmware skew surfacing as an intermittent collective hang.

A strong answer

A typical situation: 64 nodes arrive, a technician installs the driver and CUDA on all of them in an afternoon, and the first multi-node training job hangs intermittently after twenty minutes. Nobody looks at firmware, because the symptom is a NCCL timeout and firmware is six layers away from where the error appeared.

The chain, and the check that proves each link:

Two dependencies in that table are the ones that catch people:

the fabric manager needs the InfiniBand stack
  on DGX and HGX B200 and B300 systems the NVSwitches are reached through a bridge device,
    and the host control plane goes through the OFED or MOFED driver
  so installing the GPU driver and the fabric manager without the InfiniBand packages leaves
    the fabric manager failing to start
  the symptom is not an error about InfiniBand. It is nvidia-smi topo -m showing no NVLink
    between GPUs, which reads as a hardware problem

the fabric manager version must match the driver version
  a mismatch does not crash. The service refuses to configure the fabric, GPUs present as
    isolated, and a tensor-parallel job runs correctly and slowly
  this is the worst class of fault: correct output, wrong performance, no error
sanity: both of these produce "the GPUs work but multi-GPU is slow", which is why layer 5 has
        its own explicit check rather than being assumed from a successful install

Cluster Bring-Up: Firmware, Drivers and the Stack covers the chain in more depth. Node Health Checks and Burn-In covers what layer 9 should be watching once it exists.

The validation ladder, and what each rung costs to skip:

1. per node:    nvidia-smi, ibstat, nvidia-smi topo -m           seconds per node
2. one node:    nccl-tests all_reduce_perf across its 8 GPUs
                expect near the NVLink bus bandwidth; a low number here means layer 5
3. two nodes:   nccl-tests across a node pair
                close to the single-node number means GPUDirect RDMA is active
                roughly half means it is not, which points at layer 6
4. one rail:    pairwise bandwidth on each rail, all 8
                catches miscabling and a rail wired to the wrong leaf
5. full unit:   nccl-tests across all 512 GPUs, then a sustained job for 48 to 72 hours

the cost of skipping to step 5
  a fault at 512 GPUs has 512 nodes and 1,024 links as candidate causes
  the same fault at step 2 has one node
  and the cluster costs 512 x 24 x 2.5 = $30,720 per day to sit unusable while you narrow it
sanity: the whole ladder is a few hours of technician time against a five-figure daily idle
        cost, so it pays for itself the first time it catches anything at all

The pinning discipline, which is what keeps the chain stable:

• One golden image with driver, OFED, fabric manager, CUDA and NCCL versions fixed inside it, and provisioning that reimages rather than upgrading in place.
• Unattended upgrades off. A kernel that updates itself breaks the GPU driver at the next reboot, on one node, at a time nobody chose.
• Driver and fabric manager pinned together, because their versions must match.
• A firmware change record, because a firmware rollout is the most likely cause of a fleet-wide performance change and every such investigation starts by asking what changed.

The reversal condition: when nodes arrive from a vendor pre-imaged and validated, most of this collapses into verification rather than installation, and insisting on rebuilding from bare metal wastes days. The right move then is to verify the layers rather than redo them: check that firmware versions match your fleet standard rather than the vendor's, confirm the driver and fabric manager versions match each other, and run the validation ladder. Where this bites is that a vendor's image is pinned to the vendor's versions, so the first fleet-wide upgrade still needs the golden-image discipline, and building it later under time pressure is worse than building it now.

What interviewers probe next

• "What does a firmware skew look like?" Not a firmware error. An occasional NCCL hang or an XID during a large collective, which is why the inventory diff at layer 1 exists.
• "How do you know GPUDirect is on?" Two-node nccl-tests bandwidth close to the single-node number, and the NCCL INFO log naming the GPUDirect path. Roughly half the expected bandwidth means it is off.
• "Why open kernel modules?" They are the supported path on Blackwell parts, and mixing module types across a fleet is a version-skew problem by another name.
• "What do you automate?" All of it, into an image plus a provisioning run, with the validation ladder as a scripted acceptance step rather than a manual checklist.

Common mistakes

• Installing the driver and CUDA and treating bring-up as finished.
• Omitting the InfiniBand packages on Blackwell systems, leaving the fabric manager unable to start.
• Running a fabric manager version that does not match the driver, which yields correct results at wrong performance.
• Jumping straight to a full-scale test, so any fault has hundreds of candidate causes.
• Leaving unattended upgrades enabled, which breaks one node at an unpredictable time.

Key takeaways

• Nine layers in order: firmware, OS, GPU driver, InfiniBand stack, fabric manager, peer memory, CUDA and NCCL, containers, telemetry.
• On DGX and HGX B200 and B300, the fabric manager needs OFED or MOFED plus libibumad3 and infiniband-diags, and its version must match the driver.
• A fabric manager problem shows as correct output at wrong performance, with no error, which is why layer 5 has its own check.
• Validate with nccl-tests at one node, two nodes, one rail, then full scale; two-node bandwidth near single-node proves GPUDirect is active.
• Pin everything in a golden image, disable unattended upgrades, and keep a firmware change record.