TL;DR: An XID is a driver-reported error identifier, not a complete root-cause diagnosis. Match the code and context to the applicable NVIDIA catalog, perform its recovery action, and investigate recurrence separately. Some faults often originate in applications, but a familiar code does not prove the hardware or driver is healthy.
How to approach it
Record the first relevant event, device identity, job, software versions and surrounding logs. Look up the code's applicability and recovery guidance, then establish the affected scope. Keep automated recovery bounded and preserve evidence for the investigation.
A strong answer
A job fails with XID 31 after a new kernel is deployed. An application memory-access bug is a plausible explanation. It is still a hypothesis: reproducing under a diagnostic tool, comparing versions and examining device history are more useful than declaring the hardware innocent from the number alone. GPU Failure Modes and XID Errors provides the vocabulary for that investigation.
NVIDIA's XID catalog distinguishes immediate recovery from investigatory actions and identifies applicability across GPU generations. For example, XID 31 denotes a memory page fault; application bugs are common causes, but driver or hardware bugs are also possible. XID 79 means the driver cannot access the GPU over PCIe and calls for bare-metal restart guidance plus investigation of recurring faults. Use the current catalog for other codes rather than applying a memorized three-bucket table to every generation.
A useful runbook groups actions, while retaining the code-specific details:
| Decision | Evidence and boundary |
|---|---|
| Restart or debug an application | Applicable guidance, reproducible job behavior, diagnostic evidence |
| Drain and reset supported scope | Recovery state says reset is needed; affected clients are coordinated |
| Restart host or quarantine | Device/path unavailable or recovery fails; protect other allocations |
| Escalate to support or repair | Recurrence, diagnostics and vendor criteria support further action |
ECC, Row Remapping and Memory Errors adds the memory-recovery distinction. A contained error and an uncontained error have different impact, and pending remapping is not interchangeable with remapping failure. Recovery flags and device generation matter. An uncorrectable event does not automatically mean every GPU on the node requires replacement.
Keep the device UUID, PCI address, timestamp, driver version and job attempt with each event. These let you connect an error to the workload and hardware that were present at the time. One fault can produce several XIDs and watchdog timeouts on other ranks, so count the incident separately from its log lines. Check the exported metric too: a field containing the latest error code cannot be treated as an event counter.
Here is a synthetic deduplication example:
One device incident produces 12 driver log lines
The same incident is also exported to 2 collectors
Raw records = 12 × 2 = 24
Correlated incident count = 1, with all 24 records retained as evidence
Sanity: deduplication reduces notifications, not the underlying evidence.
Automation should create an incident identity, apply an idempotent admission block when required, notify affected workload control and execute a bounded recovery path. Before returning capacity, verify supported diagnostics and representative useful work. If the controller is uncertain whether a reset completed, reconcile observed state rather than issuing an unlimited loop of destructive retries.
For an apparent application fault, preserve a minimal reproducer and use the recommended debugging workflow. A failure following the job across nodes supports an application or shared-software explanation; a failure following one device supports a device or local-path explanation. Neither single comparison is conclusive because workload, driver and hardware interactions can matter. Repeat controlled tests and escalate unresolved cases.
Choose paging from impact, risk and required human action. A recoverable isolated event may need a ticket; repeated failed recovery or threatened data correctness may need immediate attention. A fixed code-to-page rule without job and capacity context can both over-page and miss the event that matters.
Reverse an initial application diagnosis if diagnostics, recurrence on one device or a version-controlled comparison contradict it. Reverse a hardware-replacement proposal if the issue is resolved by a supported software fix and acceptance tests pass. Keep the action policy versioned with its source and validation date.
What interviewers probe next
- "Does XID 31 prove a user bug?" No; it guides a debugging workflow that also considers driver and hardware causes.
- "Why keep the first event?" Later errors can be consequences rather than the initiating fault.
- "Should every event trigger a reboot?" No; follow applicable recovery scope and verify the result.
Common mistakes
- Treating error-code families as certain root causes.
- Counting repeated log lines as independent hardware failures.
- Returning capacity after a reset without a workload-level check.
Key takeaways
- XID 31 can have application, driver or hardware causes.
- Twenty-four records can describe one incident in the stated duplicate-collector example.
- Separate immediate recovery, investigation and notification policy.