A new GPU node arrives. What do you run before it serves a customer, and what does each test actually catch?

Infant mortality is real and diagnostics that run for a minute do not find it. The five-stage acceptance suite with what each stage catches and how long it takes, the pass criteria as numbers rather than judgments, and the sustained-load stage that finds the faults everything else misses.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Node Health Checks and Burn-InThe cheapest failure is the one caught before a job starts. A GPU fleet runs three kinds of check: a burn-in of days on new or repaired hardware to shake out the parts that die young, a pre-flight suite of minutes before a node is handed to a job, and continuous checks during the job. Each test has an expected number (a GEMM within a few percent of the fleet median, an all-reduce at rated bus bandwidth, a NIC at line rate, a memory test with zero errors), and a node that misses any of them goes to quarantine. This page gives the suites, the numbers, and the arithmetic that justifies the time they cost.

Foundational

🖧 Hardware & Cluster Build-Out

Burn-In and Acceptance TestingNew hardware fails early or it fails late, and burn-in exists to move the early failures before the cluster is handed over rather than after. A proper acceptance test runs every layer under sustained load for days, compares every node against its siblings rather than against a specification, and produces a signed number the buyer and the vendor both agree on. The comparison is the important part: identical hardware running identical work should produce identical numbers, and the outliers are the finding.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow would you detect that GPUs are thermally throttling, and what is the right response when they are?Next in this trackWhat do NVLink errors look like in telemetry, when is a link degrading rather than broken, and when do you drain the node?Next in this trackWrite the postmortem for a training run that lost twelve hours. What goes in it, and what makes the action items stick?

FEDITOR'S NOTE

Scored on the staged suite with a distinct fault class per stage, on numeric pass criteria rather than 'looks fine', and on sustained load being the stage that catches marginal hardware.

DISCUSSION · 0

No comments yet — be the first to share your approach.