Advanced

Node Health Checks and Burn-In

The cheapest failure is the one caught before a job starts. A GPU fleet runs three kinds of check: a burn-in of days on new or repaired hardware to shake out the parts that die young, a pre-flight suite of minutes before a node is handed to a job, and continuous checks during the job. Each test has an expected number (a GEMM within a few percent of the fleet median, an all-reduce at rated bus bandwidth, a NIC at line rate, a memory test with zero errors), and a node that misses any of them goes to quarantine. This page gives the suites, the numbers, and the arithmetic that justifies the time they cost.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

DCGM and GPU Telemetry GPU Failure Modes and XID Errors NVLink and Fabric Faults Slurm for AI Clusters Incident Response for GPU Fleets Training Uptime and Interruption Statistics

LESSONS THAT TEACH THIS

Training at ScaleAt ten thousand devices, failure is a line in the schedule16 min→Training at ScaleA loss curve that goes wrong is a systems bug until proven otherwise16 min→Training at ScaleOne slow rank sets the pace for every other rank15 min→

PRACTICE THIS IN REAL QUESTIONS

Kubernetes, Slurm & GPU SchedulingWhat do you run on a GPU node before you let a job land on it, how long does it take, and what happens on failure?→GPU Fleet Reliability & ObservabilityA new GPU node arrives. What do you run before it serves a customer, and what does each test actually catch?→Hardware, Cabling & Cluster Build-OutA node fails burn-in twice and passes on the third attempt. Do you accept it?→Hardware, Cabling & Cluster Build-Out512 GPUs are installed and the vendor wants sign-off. Design the acceptance test.→Networking, Interconnects & StorageHow do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?→GPU Fleet Reliability & ObservabilityDesign the controller that decides whether a GPU node is healthy. What are its states, and how do you tune its false-positive rate?→