Burn-In and Acceptance Testing
New hardware fails early or it fails late, and burn-in exists to move the early failures before the cluster is handed over rather than after. A proper acceptance test runs every layer under sustained load for days, compares every node against its siblings rather than against a specification, and produces a signed number the buyer and the vendor both agree on. The comparison is the important part: identical hardware running identical work should produce identical numbers, and the outliers are the finding.
TL;DR: Run the cluster hard for days before accepting it, and judge nodes against each other rather than against a datasheet. Failure rates over a component's life follow a bathtub: a raised rate early from manufacturing and handling defects, a low flat middle, a rising end from wear. Burn-in is the deliberate consumption of the early part on the vendor's account rather than on a training run's. The tests that matter run at four levels: per-GPU stress with the vendor diagnostic, per-node collectives, per-rail fabric bandwidth, and a full-scale sustained job with thermal and power at steady state. Acceptance criteria are written before the test, expressed as numbers with tolerances, and include an explicit outlier rule, because on 512 supposedly identical GPUs the useful finding is almost never a hard failure. It is the four GPUs running 6 percent slower than the other 508.
The bathtub, and what burn-in buys
failure rate over a component's life
early raised, from manufacturing defects, shipping damage, bad seating, marginal solder
middle low and roughly constant; the corpus anchor is about 2 x 10^-5 per GPU-hour
late rising, from wear: thermal cycling, HBM degradation, fan and pump wear
what a 72-hour burn-in on 512 GPUs consumes
GPU-hours = 512 x 72 = 36,864
at the steady-state rate that predicts 36,864 x 2e-5 = 0.7 failures
observed counts during burn-in are routinely several times that, and the excess is exactly
the infant mortality being drained
sanity: if burn-in finds roughly the steady-state number and no more, either the batch is
unusually good or the test is not stressing the parts, and the second is more likely
Training Uptime and Interruption Statistics has the steady-state figure and where it comes from. The burn-in argument is that the same failures cost far more later: a defect found on day 3 is a vendor replacement, and the same defect found on day 40 is a training run losing work and an on-call engineer at 2 a.m.
The four levels, and what each catches
| Level | Test | What it catches |
|---|---|---|
| Per GPU | Vendor diagnostics at the deepest level, plus a memory stress pass | HBM errors, marginal parts, cards that throttle early |
| Per node | Single-node all-reduce over 8 GPUs; a dense compute loop at sustained power | NVLink faults, a fabric manager problem, one weak GPU in a node |
| Per rail | Pairwise bandwidth on each of the 8 rails, node to node | Miscabling, a bad transceiver, a rail wired to the wrong leaf |
| Full scale | A real training job at full size for 48 to 72 hours | Thermal and power at steady state, fabric congestion, anything that only appears hot |
Compare against siblings, not against the datasheet
This is the technique that separates a real acceptance test from a checklist. The datasheet gives an upper bound that nothing quite reaches, so "within spec" is a weak claim. Identical parts running identical work, on the other hand, should agree closely, and the spread is informative.
example: single-node all-reduce bus bandwidth across 64 nodes
median across nodes: a number, whatever it is
acceptance rule: every node within 3% of the median
what an outlier means
one node 6% low usually one weak NVLink or one GPU clocking lower
four nodes 6% low often a rack: check the cooling and the power feed for that rack
one rail 15% low cabling, a transceiver, or the wrong leaf
the same rule applied to
GPU clock under sustained load, per GPU
power draw at a fixed workload, per GPU
HBM temperature at steady state, per GPU
step time in the full-scale job, per rank
sanity: on 512 identical GPUs the spread should be small, so a distribution with a tail is
a finding even when every part is inside its specification, and it is a finding you
can only make by comparing parts to each other
Writing acceptance criteria that mean something
- Numbers with tolerances, agreed before the test. "Bus bandwidth per node within 3 percent of the fleet median" is testable; "performs as expected" is not.
- A duration. Thermal problems need hours at steady state, and a 20-minute test finds none of them.
- An outlier clause, so a fleet that passes on averages and has a tail is not accepted.
- A defect budget. Some early failures are normal; agree in advance how many replacements are covered and how quickly, rather than negotiating it while the cluster is half-broken.
- The environment recorded. Inlet water temperature, hall temperature and power feed configuration, because a test passed in winter with the doors open is not a test.
ECC, Row Remapping and Memory Errors covers the memory-specific criteria, which deserve their own line: a part that has already consumed a large share of its remapping capacity during burn-in should be replaced rather than accepted, even though it currently works.
What interviewers are listening for
The sibling comparison, above everything else. A candidate who says "we ran the vendor diagnostics and everything passed" has described a checklist. A candidate who says "we compared per-node bus bandwidth against the fleet median and rejected anything more than 3 percent off, which found four nodes in one rack that turned out to share a cooling problem" has described an acceptance test. The second signal is duration and thermal steady state. The third is knowing why burn-in exists at all, which is the bathtub curve and moving the early failures onto the vendor's account.
Key takeaways
- Burn-in exists to consume infant mortality before handover; the steady-state rate of about 2 x 10^-5 per GPU-hour predicts 0.7 failures in a 72-hour run on 512 GPUs, and finding several times that is normal and is the point.
- Four levels: per GPU diagnostics, per node collectives, per rail bandwidth, then 48 to 72 hours at full scale.
- Judge nodes against the fleet median rather than the datasheet, with a stated tolerance such as 3 percent.
- Write acceptance criteria as numbers with tolerances, a duration, an outlier clause, a defect budget and a recorded environment.
- A part that has consumed much of its row-remapping capacity during burn-in should be replaced even though it currently works.