Burn-In and Acceptance Testing
Acceptance testing checks that a new cluster meets agreed performance and reliability criteria. Sustained tests and comparisons between nodes help reveal defects before production.
TL;DR: Agree on test workloads, duration and tolerances before handover. Exercise GPUs, nodes, network paths and the full cluster, recording thermal and power conditions. Compare similar nodes to find outliers, then investigate the cause. Burn-in can expose early defects, but a passing test does not guarantee future reliability.
The bathtub, and what burn-in buys
The bathtub curve is a model of changing failure rates: some defects appear early, a lower-rate period follows, and wear can raise the rate later. Actual equipment populations need not follow that shape neatly. Burn-in exposes hardware to controlled load so faults can be investigated before production.
illustrative constant-rate calculation
512 GPUs x 72 hours = 36,864 GPU-hours
assumed rate = 2e-5 events per GPU-hour
expected events = 36,864 x 2e-5 = 0.737
sanity: an expected count below one does not require a test to find a failure.
Training Uptime and Interruption Statistics explains the historical data behind fleet-rate estimates. Do not treat that rate as a measured infant-mortality rate for a new delivery. Acceptance should follow the agreed test results, not an expectation that burn-in must find a certain number of bad parts.
The four levels, and what each catches
| Level | Test | What it catches |
|---|---|---|
| Per GPU | Vendor diagnostics at the deepest level, plus a memory stress pass | HBM errors, marginal parts, cards that throttle early |
| Per node | Single-node all-reduce over 8 GPUs; a dense compute loop at sustained power | NVLink faults, a fabric manager problem, one weak GPU in a node |
| Per rail | Pairwise bandwidth on each of the 8 rails, node to node | Miscabling, a bad transceiver, a rail wired to the wrong leaf |
| Full scale | A real training job at full size for 48 to 72 hours | Thermal and power at steady state, fabric congestion, anything that only appears hot |
Keep workload, duration, tolerance and environmental conditions with every result.
- GPU
Run the supported diagnostics and memory stress checks.
- Node
Exercise the local GPU fabric and sustained device load.
- Rail
Compare the expected network paths and investigate asymmetry.
- Full cluster
Run the agreed workload long enough to evaluate steady-state power, cooling and communication.
Yes
- Advance or sign off
Advance to the next scope; after full-scale acceptance, retain the signed measurements.
No
- Isolate and investigate
Identify the failing path or component, correct it, then rerun the affected checks. Do not assume every outlier needs hardware replacement.
Compare against siblings, not against the datasheet
This is the technique that separates a real acceptance test from a checklist. The datasheet gives an upper bound that nothing quite reaches, so "within spec" is a weak claim. Identical parts running identical work, on the other hand, should agree closely, and the spread is informative.
example: single-node all-reduce bus bandwidth across 64 nodes
median across nodes: a number, whatever it is
acceptance rule: every node within 3% of the median
what an outlier means
one node 6% low usually one weak NVLink or one GPU clocking lower
four nodes 6% low often a rack: check the cooling and the power feed for that rack
one rail 15% low cabling, a transceiver, or the wrong leaf
the same rule applied to
GPU clock under sustained load, per GPU
power draw at a fixed workload, per GPU
HBM temperature at steady state, per GPU
step time in the full-scale job, per rank
sanity: on 512 identical GPUs the spread should be small, so a distribution with a tail is
a finding even when every part is inside its specification, and it is a finding you
can only make by comparing parts to each other
Writing acceptance criteria that mean something
- Numbers with tolerances, agreed before the test. "Bus bandwidth per node within 3 percent of the fleet median" is testable; "performs as expected" is not.
- A duration. A short test may catch immediate thermal faults but miss problems that appear during sustained operation.
- An outlier clause, so a fleet that passes on averages and has a tail is not accepted.
- A defect budget. Some early failures are normal; agree in advance how many replacements are covered and how quickly, rather than negotiating it while the cluster is half-broken.
- The environment recorded. Inlet water temperature, hall temperature and power feed configuration, because a test passed in winter with the doors open is not a test.
ECC, Row Remapping and Memory Errors covers the memory-specific criteria, which deserve their own line: a part that has already consumed a large share of its remapping capacity during burn-in needs evaluation against the vendor’s health and replacement criteria before acceptance.
What interviewers are listening for
Describe what the test must establish and how you would investigate a failing result. Fleet comparisons can reveal problems that averages hide, but a threshold needs to account for expected variation and test conditions. If you use a hypothetical cooling fault as an example, label it as one rather than presenting it as your operating experience.
Key takeaways
- Burn-in is a chance to find defects before production, not a guarantee that early failures have been exhausted.
- Test at GPU, node, fabric and full-cluster scope.
- Compare similar nodes under controlled conditions and investigate performance outliers.
- Agree on tolerances, duration, environment and replacement terms before testing.
- Apply vendor health and replacement criteria to memory errors and row-remapping results.