TL;DR: Follow the platform's supported dependency order, validating each layer before expanding the test. Inventory and device discovery come first; correct computation, interconnect behavior, container access and recovery establish whether the nodes are ready for jobs.
How to approach it
Identify the exact node platform and its qualified software recipe. Explain what each layer needs from the preceding layers and what test could reveal a failure there. Scale from one device to a node, a node pair and the intended job group while preserving a known-good comparison.
A strong answer
An installation can complete successfully while its first distributed workload fails. The driver may discover every GPU, yet a container lacks a library or the interconnect cannot initialize. Bring-up should narrow these possibilities progressively, with saved inventories and results at each stage.
Start from the vendor's installation and compatibility guidance for the actual system. The Fabric Manager guide (Fabric Manager is the NVIDIA service that configures NVSwitch-connected GPUs into one NVLink fabric) describes generation-specific dependencies and configurations. DGX/HGX and systems have requirements involving their NVSwitch control path and InfiniBand-related packages. Do not apply a recipe for an earlier platform without checking those requirements and the supported driver/Fabric Manager pairing.
| Stage | What to establish |
|---|---|
| Hardware and firmware | Expected devices, cabling and qualified component versions |
| OS and drivers | Supported kernel/modules, device discovery and a correct compute workload |
| Local fabric | Required services initialized; expected connectivity and working peer communication |
| Scale-out transport | NIC/link health, GPU-to-NIC mapping and the intended data path |
| Runtime and libraries | Compatible container access, CUDA and collective behavior |
| Scheduler and telemetry | Correct allocation, health gating, complete metrics and tested alerts |
This is a dependency map rather than a universal package-install order. An image may already contain several layers, and kernel modules may depend on the chosen networking stack. Verify an accepted vendor image before deciding to rebuild it. Keep a versioned, reproducible recipe and a controlled update process, including security fixes; avoiding unplanned kernel changes is not a reason to leave a fleet unpatched indefinitely.
Cluster Bring-Up: Firmware, Drivers and the Stack explains the component relationships. Device visibility from nvidia-smi establishes discovery, not numerical correctness. A topology report shows a connectivity view, not proof that every fabric service is healthy or every path performs correctly. Check service state and logs, then execute workloads that exercise the relevant paths.
GPUDirect RDMA also has more than one supported setup. NVIDIA's GPU Operator RDMA documentation distinguishes DMA-BUF and legacy nvidia-peermem prerequisites. Loading one module is not the universal definition of a working direct path. Confirm the selected mechanism and transport using the installed stack's diagnostics and representative transfers.
Build a validation ladder with an expected result at each level. A single-device correctness test isolates arithmetic and runtime access. A node-level collective exercises local connectivity. A two-node test adds NICs and scale-out links. Further tests should cover rails, leaves and the intended group size before sustained application runs. Include representative message sizes, rank mapping and concurrent traffic; an isolated bandwidth maximum is only one part of acceptance.
Do not diagnose the transport from a fixed ratio between single-node and two-node bandwidth. Those tests use different paths and may choose different algorithms. A low result can come from placement, arrival skew, link configuration, CPU staging or other contention. Use the timeline, logs and counters to choose the next test, then check whether it changes useful application throughput.
An illustrative fleet of 512 GPUs in eight-GPU nodes has 64 nodes. A failure seen at full scale therefore has a much larger search space than one reproduced on a single node, but not 512 nodes. At an assumed $2.50 per GPU-hour, one day of idle fleet capacity corresponds to $30,720. This comparison motivates staged diagnosis; it does not justify skipping correctness checks to save a few minutes.
Before admitting production work, test recovery as well as speed. Drain a designated test node, exercise the supported replacement path and restore a checkpoint. Confirm that health gates prevent a failed node from being scheduled, and that an alert reaches its intended destination. Plan disruptive acceptance tests with the responsible operators. Node Health Checks and Burn-In covers how these checks continue after commissioning.
If a test fails, stop expansion at that boundary and compare with a known-good peer. Preserve the evidence before resetting or reimaging. If the vendor's prequalified image passes the same requirements, verification may be enough; a custom rebuild needs a concrete benefit and its own maintenance plan.
What interviewers probe next
- "How do you prove GPUDirect is used?" Combine the configured transport's diagnostics with targeted transfers; a bandwidth ratio alone is insufficient.
- "Does firmware skew explain every NCCL timeout?" No. It is a hypothesis to compare against the inventory and failure evidence.
- "What should be automated?" Inventory, provisioning, acceptance results and health admission, with explicit exceptions and rollback.
- "When can a node rejoin?" After the failed path and representative workload pass the defined checks.
Common mistakes
- Declaring success because packages installed or devices appeared.
- Using the same dependency recipe for every GPU generation.
- Treating topology output as proof of fabric readiness.
- Applying unplanned updates without a tested rollback path.
Key takeaways
- Test discovery, correct execution and useful distributed work separately.
- A 512-GPU fleet with eight GPUs per node contains 64 nodes.
- Expand the test only after the smaller failure domain passes.