TL;DR: Agree on the delivered configuration, test procedures and acceptance criteria before delivery. Test individual GPUs, whole nodes, fabric paths and a representative full-scale workload. Combine absolute requirements with peer comparisons, record the environment and investigate outliers. A 48–72-hour soak can be a chosen test window, but neither that duration nor a universal 3% tolerance proves reliability.
How to approach it
Define what the system must deliver and how each requirement will be measured. Stage tests so failures remain diagnosable, then specify the evidence, remediation and retesting needed before sign-off. Treat durations and tolerances as justified choices rather than industry constants.
A strong answer
Suppose 64 eight-GPU nodes have been installed. The vendor reports that diagnostics pass, while the customer needs a cluster that can sustain a particular training workload. Those are related requirements, but they are not the same test. Burn-In and Acceptance Testing explains why component health and application readiness both belong in the acceptance plan.
Write a testable specification
Record the server and GPU SKUs, serials, topology, firmware, drivers, runtime, power settings and cooling conditions. Name the benchmark versions, message sizes, precision, workload data and placement. State what counts as pass, fail, unavailable or skipped. An unsupported test that did not run must not appear as a passing result.
Use four stages, with the chosen limits attached to each:
| Stage | What to exercise | Evidence to retain |
|---|---|---|
| Per GPU | Supported compute, memory and transfer diagnostics | Per-device results, error history, remapping state and skipped-test reasons |
| Per node | GPU peer paths, sustained compute/memory work and collectives | Correctness, bandwidth/latency curves, clocks, power, temperatures and limiting reasons |
| Fabric | Expected GPU/NIC mapping, relevant pairs, rails and shared paths | Negotiated modes, traffic/counter deltas and performance against the topology requirement |
| Full system | Representative sustained workload and agreed recovery cases | Useful throughput, step/latency distribution, interruptions, environment and recovery results |
NVIDIA's DCGM diagnostic documentation makes test availability dependent on release, hardware, plugins and other prerequisites. Select a supported suite with the supplier; do not assume the highest level is identical on every GPU or substitutes for the vendor's field-diagnostic process.
Use an absolute floor and an outlier check
A known-good configuration or agreed workload target establishes the minimum acceptable result. Peer comparison then finds unusually weak devices or paths. A fleet median alone cannot detect a common misconfiguration that slows every node.
For example, suppose the measured peer median is 100 units of throughput and a previously justified tolerance is 3%. The relative lower threshold is 97. A node delivering 94 fails that comparison. If the entire fleet instead delivers 80 against an agreed floor of 95, a narrow spread around 80 does not make it acceptable.
Illustrative throughput units; thresholds must be agreed and reproducible.
Peer outlier
- Median 100; node 94
A 3% lower tolerance gives 97. The node needs investigation.
Common shortfall
- All nodes 80; required floor 95
A tight peer spread does not satisfy the independent system requirement.
Choose the tolerance from repeated reference tests, measurement noise and legitimate variation. Compare matched hardware and settings. A temperature or clock difference is a diagnostic signal, not always a defect: workload distribution, sensor placement and allowed power limits matter. “No throttling” is too vague when normal power-limit behavior may be expected; define prohibited thermal excursions and required sustained performance.
Map outliers to racks, cooling groups, firmware, manufacturing batches and fabric paths. Four weak nodes in one rack suggest a shared cause but do not prove cooling is responsible. Reproduce the result, inspect signals and test the hypothesis. Hard failures remain important even if the spread is otherwise excellent.
Choose duration for coverage, not a desired failure count
Run long enough to observe thermal stabilization, sustained load, relevant workload phases and agreed recovery behavior. A 72-hour full-cluster run is one possible acceptance window. It is not a universal prescription, and a shorter test can still find real problems.
The exposure arithmetic is straightforward:
512 GPUs × 72 hours = 36,864 GPU-hours
If, purely for illustration, independent failures follow a constant rate
of 0.00002 per GPU-hour:
expected count = 36,864 × 0.00002 = 0.73728
probability of zero = exp(-0.73728) ≈ 47.8%
sanity: zero failures is quite plausible under this assumed model.
This rate is not a measured fleet baseline. Early defects, correlated outages and test coverage can violate the model. Finding no failures does not show that the test was too weak; finding several does not automatically demonstrate useful “infant mortality removal.” Inspect the actual failures and the tests that ran.
Record coolant and air conditions, flow where relevant, power limits and sustained temperatures. A run at mild conditions may not establish performance at the specified worst-case environment. Confirm the required operating envelope through the supplier's approved test and qualification evidence rather than improvising hazardous fault tests.
Make remediation and retesting explicit
The acceptance package should identify unresolved defects, excluded capacity, replacement timelines, retest scope and who can approve an exception. Agree these with procurement and the supplier; the engineering test plan alone does not establish contract enforceability.
Define memory-health treatment using supported diagnostics and vendor criteria. ECC, Row Remapping and Memory Errors explains why a count needs context: pending remaps, failures and bank-level availability can matter more than a single percentage. Do not invent one replacement threshold for all parts.
Retest replaced or repaired components at the affected levels, then repeat integration tests when the change could affect the cluster result. Keep artifacts linked to hardware serials and configuration. Sign off only on the capacity and requirements supported by the evidence; an average should not conceal an unresolved required-node failure.
For a smaller delivery, peer comparison still has value, but an independent reference is even more useful. Four nodes have a perfectly valid median; the limitation is that a small group provides less evidence about normal variation and shared defects. The absolute reference remains useful at every fleet size.
What interviewers probe next
- “Why 3%?” It is an example that needs repeatability evidence, not a universal hardware tolerance.
- “What if every node passes diagnostics but the job is slow?” Investigate the application, topology and system target; passing component tests does not satisfy a separate performance requirement.
- “Does a clean soak prove reliability?” It establishes behavior over that exposure and test coverage, not a lifetime guarantee.
Common mistakes
- Relying only on a fleet median or only on a theoretical datasheet peak.
- Counting unavailable diagnostics as passed.
- Requiring an arbitrary number of burn-in failures.
- Accepting replacement hardware without an appropriate retest.
Key takeaways
- Combine component, node, fabric and full-workload evidence.
- Use both an absolute performance requirement and a justified outlier tolerance.
- A 72-hour run provides 36,864 GPU-hours; report what it tested and what remains unproven.