TL;DR: This error means the GPU has stopped responding on the PCIe bus entirely: the driver's requests time out, the device is no longer enumerable, and
nvidia-smieither omits it or reports it as fallen off. There is no software recovery, because resetting a device requires talking to it. The host must reboot so the bus is re-enumerated, which makes the whole node unavailable, not just the one GPU. The physical causes are power delivery under transient load, thermal events, a marginal PCIe link or riser, and the device itself failing. The response is mechanical: notify the running job, cordon the node, reboot, run the health suite, and return it to the pool if it passes. What separates a competent platform from an incompetent one is the policy after that: a single occurrence on a node that then passes a full diagnostic is a transient and the node goes back to work; the same node doing it three times in a month is a hardware return regardless of how well it tests in between.
How to approach it
Say what the error physically means and why that rules out software recovery, since candidates often propose a driver reset. Then the causes, ordered by how often they turn out to be the answer. Then the automated response, and then the repeat policy, which is where the judgment lives. Close with why the failure takes the node rather than the GPU.
A strong answer
A typical situation: a node drops a GPU during a training run at 3am. The on-call engineer reboots it, nvidia-smi shows eight healthy GPUs, the diagnostics pass, and the node goes back into the pool. This happens four more times over three weeks on the same node, each time handled by a different engineer, and nobody notices the pattern because each event was closed as a transient.
What the error means:
normal operation the driver communicates with the GPU over PCIe: configuration space, memory
mapped registers, DMA
this fault those accesses stop being answered. The driver's reads return all ones, its
writes go nowhere, and the device disappears from the bus
observable as nvidia-smi reports the device as fallen off the bus, or omits it entirely
the kernel log shows the fault code and often preceding PCIe errors
why no software fix a reset is a command sent to the device. A device that does not answer
cannot be commanded. The PCIe hierarchy must be re-enumerated, which in
practice means a host reboot, and on some systems a full power cycle
because a warm reboot does not reset the device's power state
GPU Failure Modes and XID Errors covers where this sits in the fault taxonomy.
The causes, in rough order of how often each is the answer:
| Cause | Mechanism | Corroborating evidence |
|---|---|---|
| Power delivery under transient load | A sudden draw exceeds what the supply or the board can deliver, and the device browns out | Correlated with a job's start, or with several GPUs ramping together; power telemetry showing a dip |
| Thermal event | The device or its memory exceeds a limit and shuts down rather than throttling | Temperature history rising before the event; other GPUs in the same chassis warm |
| Marginal PCIe link, riser or connector | The link errors out and drops | PCIe replay and error counters climbing on that slot beforehand; the fault follows the slot when the GPU is moved |
| The device itself | Silicon or board failure | Everything else is clean, the fault follows the GPU when it is moved to another slot |
the diagnostic that separates the last two, and the only one that is conclusive:
move the GPU to a different slot, or a different GPU into the same slot
fault follows the GPU -> the device
fault stays with the slot -> the riser, the connector or the board
cost: a maintenance window and physical access, which is why it is reserved for a node that
has failed repeatedly rather than done on the first event
The automated response:
what the automation must do and what it must not:
must notify the job before cordoning, so it can checkpoint if it is able. A job killed
without warning loses more than one warned a few seconds ahead
must keep a per-node counter with a time window, because the decision depends on history
must record the event with its timestamp, the job, and the preceding telemetry, so a later
pattern is visible
must not return the node silently on a pass, with no record, which is how the same node
fails five times and nobody connects them
The repeat policy, which is the judgment this question is really about:
one event, health suite passes transient. Return to the pool. Count it.
two events in 30 days ticket for physical inspection at the next window,
keep the node in service but avoid scheduling the
longest runs on it
three events in 30 days retire from the pool regardless of test results.
A node that passes every test and fails every few
days is a node whose failure mode the tests do not
reproduce
why a count rather than a test result: this fault is intermittent by nature, and a diagnostic
run for thirty minutes on a healthy-looking device
will pass. The history is better evidence than the
test, which is the general rule for intermittent
hardware
the arithmetic that makes three events conclusive:
one node, 8 GPUs, 30 days = 8 x 30 x 24 = 5,760 GPU-hours
observed rate with 3 events = 3 / 5,760 = 5.2e-4 per GPU-hour
fleet baseline = 2e-5 per GPU-hour
ratio = 5.2e-4 / 2e-5 = 26x the fleet rate
P(3 or more events by chance) at the baseline rate, expected 5,760 x 2e-5 = 0.115 events:
Poisson with mean 0.115 gives P(k >= 3) = 2.3e-4
sanity: a one-in-four-thousand coincidence, on a node you have already looked at twice, is
not a coincidence. The count is conclusive without any diagnosis succeeding
Node Health Checks and Burn-In is the suite in the flow; Incident Response for GPU Fleets covers the notification and ticketing path.
The reversal condition: on a node hosting several independent single-GPU workloads rather than one multi-GPU job, the calculus changes: the reboot takes down seven healthy GPUs serving seven unrelated tenants, so the cost of the response is higher and the case for tolerating a degraded node with the failed GPU excluded is stronger. Some platforms do exactly that, marking the device unavailable and continuing on the remaining seven until a maintenance window. That is defensible for inference and wrong for training, where the node's GPUs are one unit and a seven-GPU node is not schedulable anyway.
What interviewers probe next
- "Why not just reset the GPU?" A reset is a command to the device, and the device is not answering commands. That is what falling off the bus means.
- "Does a warm reboot always work?" Not always. Some cases need a full power cycle to reset the device's power state, which is why the automation should escalate rather than loop on warm reboots.
- "How would you catch it before it happens?" PCIe replay and error counters on the slot, and power and thermal telemetry, which sometimes show a trend. Often there is no warning, which is why the repeat policy exists.
- "What do you tell the customer on a multi-tenant cloud?" That the node had a hardware fault, what was done, and what the replacement path is. The event is visible to them, so silence is worse than a plain description.
Common mistakes
- Proposing a driver-level reset for a device that cannot receive commands.
- Closing each occurrence as a transient with no counter, so a repeatedly failing node keeps returning to service.
- Cordoning without notifying the running job, which loses work that a few seconds of warning would have saved.
- Swapping the GPU on the first event, when the slot is as likely a cause and the swap tells you nothing without the history.
Key takeaways
- The device has stopped answering on PCIe, so there is no software recovery: the host must reboot, and sometimes power-cycle.
- Causes in order: power delivery under transient load, thermal, a marginal link or riser, the device itself.
- The automation notifies the job, cordons, drains, reboots, runs the health suite, and increments a per-node counter.
- Act on the count, not the test: three events in 30 days retires the node regardless of passing diagnostics, since that rate is orders of magnitude above the fleet's.