TL;DR: A correctable error means the memory returned wrong bits and the error-correcting code fixed them, so the computation was right and nothing failed. The value is predictive: memory that is starting to degrade produces correctable errors before it produces an uncorrectable one, and an uncorrectable error during a training run kills the job and may corrupt a checkpoint. The device can heal itself up to a point through row remapping, which retires a failing row and substitutes a spare, but the spare capacity is finite. So the policy has three tiers. A rising correctable rate on one device opens a ticket and marks it for a drain at the next checkpoint boundary, where a reset applies any pending remaps. A pending remap that persists after reset, or a remap failure meaning spares are exhausted, means the device cannot heal and is replaced. An uncorrectable error means the device is drained immediately and returned. The whole point is to move the event from the middle of a run to a scheduled window.
How to approach it
Say what a correctable error is and why its value is predictive rather than diagnostic, because a candidate who treats it as a fault will drain nodes constantly and one who ignores it will lose runs. Explain remapping as a finite budget, since that is what determines whether the device recovers. Then the three-tier policy with the timing, which is the actual answer. Close with the numbers that make it worth doing.
A strong answer
A typical situation: a device's correctable error count has gone from a handful per week to several hundred per day over six days. Nothing has failed, every job on that node has completed correctly, and the monitoring shows the node as healthy because the errors are corrected by definition. Nine days later an uncorrectable error kills a 1,024-GPU run twenty-five minutes after its last checkpoint.
What the signal is:
correctable (single-bit) error the memory returned wrong bits, the ECC logic detected and
corrected them, and the computation proceeded correctly
uncorrectable (double-bit) more bits were wrong than the code can correct. The data is
wrong and the driver reports a fatal fault
what rising correctable rates mean physically: a cell, a row or a portion of the memory
interface is degrading. Errors become more frequent before
they become uncorrectable
what they do NOT mean: that anything computed so far was wrong. This is the point that
decides whether the response is a page or a ticket
ECC, Row Remapping and Memory Errors covers the mechanism; GPU Failure Modes and XID Errors covers the codes that carry these events.
The remapping budget, which decides whether the device can recover:
mechanism when a row accumulates errors, the device marks it for remapping. On the next
reset, the row is retired and a spare row is substituted, transparently
budget each memory die reserves a fixed number of spare rows. It is finite and per bank
states you will see:
pending row remap a remap is queued and requires a reset to take effect
row remap success a row was retired and replaced; the device healed
row remap failure no spare available for that bank; the device cannot heal further
what "cannot heal further" means: the next degradation in that bank produces uncorrectable
errors with no recourse, so the device is on borrowed time
sanity: a device with a handful of successful remaps over a year is normal and healthy;
a device with a remap failure is not, regardless of how well it is currently running
The three-tier policy, with the timing that makes it non-disruptive:
tier 1: rising correctable rate, no pending remaps
signal the device's correctable rate over a rolling window, compared against its own
baseline rather than a fleet constant, sustained over hours rather than spiking
action open a ticket, tag the device, schedule a drain at the next checkpoint boundary
timing at a 30-minute checkpoint cadence, the wait is at most 30 minutes and costs the
job nothing beyond the drain itself
tier 2: pending row remap
action drain at the next checkpoint, reset the GPU to apply the remap, run the health
suite, return to the pool if it passes
cost the reset takes minutes; the node is out for well under an hour
after watch the device: a remap that recurs quickly is a device on its way out
tier 3: remap failure, or any uncorrectable error
action cordon immediately, do not wait for a checkpoint, notify the running job so it can
checkpoint if it is able, drain, and return the hardware
reason there is no self-healing left, so the next event is a job-killing failure
sanity: tiers 1 and 2 are scheduled and cost minutes; tier 3 is unscheduled and costs the job.
The whole value of the policy is moving events from tier 3 into tiers 1 and 2
What the policy is worth:
unscheduled: an uncorrectable error mid-run
work lost = checkpoint interval / 2 + restart = 30 / 2 + 5 = 20 minutes
cluster cost = 1,024 GPUs x $2.5 per GPU-hour x (20 / 60) h = $853 per event
scheduled: a drain at the next checkpoint boundary
work lost = 0, because the job checkpoints anyway; the node's share is replaced from
the spare pool
cost = the reset and health suite, about 15 minutes of one node = 8 x 2.5 x 0.25
= $5
fleet effect, 16,384 GPUs
interruptions per day = 2e-5 x 16,384 x 24 = 7.9
memory-related share = about 17% = 1.3 per day
caught predictively, say half = 0.67 per day
saving = 0.67 x ($853 - $5) = $568 per day, about $207,000 a year
sanity: the per-event saving is modest and the case rests on frequency. That is the usual
shape of reliability work, and stating it this way is what gets it prioritized
The reversal condition: this policy assumes the fleet has spare capacity so a drained node can be replaced without shrinking the job. On a cluster running at full allocation with no spares, draining a node mid-run costs the job a restart at a smaller size, which may cost more than the risk of the uncorrectable error. There the correct policy is to record the trend, avoid scheduling new long runs on that device, and drain it at the next natural boundary between jobs. That is a worse outcome and it is the honest one when spares do not exist, which is another argument for budgeting them.
What interviewers probe next
- "Why not drain immediately on any correctable error?" They are common and mostly benign. Draining on each one would take a large fraction of the fleet out continuously and train the team to disable the policy.
- "What baseline do you compare against?" The device's own history plus a fleet-wide outlier check, since devices differ and a fleet constant either over-fires or misses.
- "Does a reset always apply the remap?" It requires a full GPU reset, which usually means no processes attached, which is why the drain has to happen first.
- "What if the errors follow the workload rather than the device?" Then it is not degradation: check whether one job's memory access pattern correlates, since a specific region being exercised harder can surface a marginal cell.
Common mistakes
- Treating correctable errors as a fault and paging on them, which drains healthy devices constantly.
- Ignoring them because they are corrected, which is how a fleet loses runs to failures it was warned about for a week.
- Comparing against a fleet-wide threshold instead of each device's own baseline.
- Draining immediately rather than at the next checkpoint boundary, which turns a free action into a costly one.
Key takeaways
- Correctable errors mean nothing failed; their value is that they precede uncorrectable errors, which do kill jobs.
- Row remapping lets a device heal until its spare rows run out; a remap failure means there is no recourse left.
- Three tiers: rising rate is a ticket and a drain at the next checkpoint, a pending remap is a drain and reset, a remap failure or uncorrectable error is an immediate cordon.
- The value is moving an event from mid-run, costing about 20 minutes of a large job, into a scheduled window costing minutes.