TL;DR: Investigate the rising rate, then use the GPU’s recovery flags and diagnostic history to decide whether it can keep running. Corrected errors are a warning to examine, not a reliable countdown to failure. If maintenance can wait for a checkpoint, that may save replaying work, but the restart still costs time.
How to approach it
Establish that the rate actually changed on the same device and metric scope. Check recovery flags, related errors and workload exposure. Then compare the risk of continuing with the complete cost of maintenance and verify the device before reuse.
A strong answer
A GPU's corrected-error counter rises much faster this week. The first question is whether the comparison is valid: was the driver reset, did the exporter change fields, did the device identity change, or did a workload begin exercising a different memory region? ECC, Row Remapping and Memory Errors explains correction; operational interpretation needs that context.
Error-correcting code (ECC) lets the memory system detect and repair certain errors. A correctable event says that this particular detected error was repaired. It cannot tell you whether unrelated software bugs or undetected errors affected the run. Use the reported correctable and uncorrectable categories rather than assuming they always mean exactly one flipped bit and two flipped bits. What a rising rate predicts depends on the device and the failure mechanism.
Same device and counter scope, with no reset:
Counter at start: 100 events
Counter after 24 hours: 280 events
Observed rate = (280 − 100)/24 = 7.5 events/hour
Prior comparable rate: 0.5/hour → observed ratio = 15×
Sanity: a reset from 280 to 0 is not negative error production; it starts a new interval.
Compare temperature, workload, allocation history and other devices under similar exposure. A workload-specific pattern does not eliminate hardware involvement: exercising a marginal location can expose a device fault. Preserve the event history rather than declaring any lifetime count harmless or any rate increase fatal.
NVIDIA's recovery and response flags distinguish pending row remapping from required recovery actions. A pending flag indicates remapping at a subsequent GPU reset; supported unaffected applications can continue in the documented conditions. Other recovery states can require reset, host reboot or drain-and-reset. Follow the state and the applicable platform guidance instead of converting every memory event into the same node-wide response.
Remapping is not an unlimited cure, but failure is not synonymous with one universal spare-row count. The RMA policy includes generation-dependent behavior and field-diagnostic validation. Use that process for replacement eligibility. Do not promise that one uncorrectable event always requires returning the hardware or that a passing reset proves permanent health.
For a trend without a mandatory immediate recovery action, open an investigation and choose a maintenance window based on workload criticality, repeat behavior and spare capacity. For an unsafe recovery state, block new admission promptly and coordinate affected jobs. A checkpoint from potentially affected state is not automatically a trusted recovery point; preserve the last known-good durable version.
A scheduled drain can avoid replaying the interval since the last checkpoint, but the job may still restart. In a hypothetical 1,024-GPU rigid job, a five-minute restart costs 1,024×5/60 = 85.33 GPU-hours. At an assumed $2.50 per GPU-hour, that is $213.33 before diagnostics and replacement-node cost. An unexpected event adding fifteen minutes of replay plus the same restart costs 341.33 GPU-hours, or $853.33. The possible saving is substantial, but it depends on actually avoiding the event and cannot be assumed for every ticket.
Measure maintenance outcomes: rate after intervention, recurrence, diagnostic results, useful time lost and false-positive removals. Quarantine repeated unexplained failures or failed recovery, and escalate with device history and reproducible evidence. Node Health Checks and Burn-In provides the acceptance layer; choose tests for the suspected failure and supported hardware.
The investigation can change the decision in either direction. A confirmed exporter change may explain the apparent rise and make a drain unnecessary. A newly reported recovery requirement may make waiting unsafe. Record the evidence behind that choice so the next operator can see why the GPU remained available or was removed.
What interviewers probe next
- "Does corrected mean the entire run is certified?" No; it describes correction of that detected event.
- "Is draining at a checkpoint zero-cost?" It can avoid replay but still incur restart and maintenance cost.
- "What decides replacement?" Applicable vendor criteria, diagnostics and the device's evidence, not a universal counter threshold.
Common mistakes
- Claiming correctable errors always precede a fatal event.
- Treating pending and failed remapping as identical.
- Ignoring restart cost in a planned-drain calculation.
Key takeaways
- 180 new events in 24 hours is 7.5/hour when counter scope is stable.
- A five-minute restart of 1,024 GPUs costs 85.33 GPU-hours.
- Use recovery state and validated evidence to choose the action.