Cluster Bring-Up: Firmware, Drivers and the Stack
Bring-up validates the dependencies between firmware, drivers, networking and the runtime. Checking each layer makes faults easier to locate.
TL;DR: Build a compatible software and firmware stack, and validate it from individual devices through multi-node jobs. Follow the platform’s dependency order: some HGX systems need the InfiniBand software stack before Fabric Manager. Record versions and test changes on a small group before rolling them across the fleet.
The dependency chain
Use the dependency order supported by the specific platform. Keep the component inventory beside the test results.
Make the devices and links usable
- Firmware and host OS
Record the BMC, BIOS, GPU, switch and NIC firmware, then install the supported OS and kernel.
- GPU driver and network prerequisites
Check the platform’s driver, kernel-module and InfiniBand requirements.
- Fabric and direct-transfer path
Initialize the supported Fabric Manager stack and verify the peer-memory or DMA-BUF path.
Make workloads and failures observable
- Runtime and collectives
Pin the toolkit and communication-library versions; test one node before adding nodes.
- Containers and scheduler integration
Verify that the test workload sees exactly the devices it requested.
- Telemetry and health gates
Check per-device coverage and demonstrate that a failed health check is reported.
The dependency order helps explain failures that appear later. A firmware mismatch may fail immediately or surface later as a collective hang or GPU error. Keeping the inventory with the test results makes that dependency easier to investigate.
What each step is, and what proves it worked
| Step | What you install | Initial validation check |
|---|---|---|
| Firmware | BMC, BIOS, CPLD, VBIOS, NVSwitch, NIC, all pinned | An inventory report showing one version per component across every node |
| OS and kernel | A pinned kernel; unattended upgrades disabled | uname -r identical fleet-wide |
| GPU driver | Data Center driver, open kernel modules on Blackwell | nvidia-smi lists every GPU at the expected clocks and power limit |
| InfiniBand stack | OFED or MOFED, libibumad3, infiniband-diags | ibstat shows every port Active at the expected rate |
| Fabric manager | nvidia-fabricmanager at the same version as the driver | The service is running and nvidia-smi topo -m shows NVLink between all GPUs |
| GPUDirect | nvidia-peermem | The module is loaded, and NCCL logs show the GDRDMA path rather than a host bounce |
| CUDA and NCCL | Toolkit and NCCL, versions pinned | nccl-tests all-reduce reaching expected bus bandwidth on one node |
| Containers | Container toolkit, device plugin or DRA driver | A test pod sees the GPUs it asked for |
| Telemetry | DCGM exporter, health checks | Metrics arriving for every GPU, and a deliberately failed check firing |
Check the driver and Fabric Manager pairing against NVIDIA’s Fabric Manager guide. An incompatible pairing can prevent initialization or affect connectivity; a slow job is not its only possible symptom. For GPUDirect, verify the supported peer-memory or DMA-BUF path. GPUDirect RDMA and GPUDirect Storage explains how to investigate host staging.
Validate at each layer, with the cost of not doing it
Start with firmware inventory, device enumeration and network port state. Then inspect the GPU/NIC topology and run communication tests at increasing scope: one node, two nodes and the intended group. Compare like-for-like message sizes and algorithms rather than expecting a benchmark to equal an advertised link rate.
A 512-GPU cluster with eight GPUs per node has 64 nodes. A failure seen only at full scale has more candidate paths and interactions than a failed local test. This is why the order helps diagnosis; it does not make any one command proof of a root cause.
The version pinning discipline
Everything above is a version, and versions drift. The practices that hold up:
- One golden image, built once, with driver, OFED, fabric manager, CUDA and NCCL versions fixed inside it, and node provisioning that reimages rather than upgrades in place.
- Controlled updates. Test kernel and driver compatibility before rollout, including required security updates. An untested kernel change can leave a node without a working driver after reboot.
- The driver and fabric manager pinned together, since their versions must match.
- A change record for firmware, so a fleet-wide change can be compared with its rollout history.
Containers, Images and GPU Cold Starts covers what runs above this line; the bring-up discipline is what makes the layer below the container reliable enough that a container problem is actually a container problem.
What interviewers are listening for
Explain the dependency order for the platform in question and what each check establishes. Device enumeration alone does not prove that NVLink, GPUDirect or collectives work correctly. Increase test scope from one GPU to one node to multiple nodes, keeping logs and version inventories so failures remain attributable.
Key takeaways
- The order is firmware, OS and kernel, GPU driver, InfiniBand stack, fabric manager, peer memory, CUDA and NCCL, containers, telemetry.
- NVIDIA's documentation states that DGX and HGX and systems need an OFED or MOFED driver plus
libibumad3andinfiniband-diagsfor the fabric manager to reach the NVSwitches. - The fabric manager version must match the driver version; verify the supported pairing and successful fabric initialization.
- Validate with
nccl-testsat one node, two nodes, then the full unit, because a fault found at full scale has hundreds of candidate causes. - Pin everything in a golden image, disable unattended upgrades, and keep a firmware change record.