Write the postmortem for a training run that lost twelve hours. What goes in it, and what makes the action items stick?

The timeline is the easy part and the detection gap is the valuable part: not what broke, but how long it was broken before anyone knew and why. A worked example with its five sections, the distinction between the trigger and the cause, and the property that separates action items that ship from ones that do not.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsAn incident on a GPU fleet is a training run that stopped, a serving endpoint burning its error budget, or a fleet-wide symptom nobody has explained yet. The response has a shape: detect, stabilize, diagnose, repair, return through the gate, write it up. The stabilizing move (drain the node, restart from checkpoint, or shift traffic) comes before the diagnosis, because a frontier run loses more per minute than any investigation is worth. This page gives the triage order, the 3am decision tree, the spare-capacity arithmetic behind drain-and-replace, and what a fleet postmortem has to contain.

Core

🧭 Ownership & JudgmentSign in

On-Call Narratives That LandEvery infrastructure loop has a round where you are asked to tell an incident story, and the interviewer is not listening for drama. They are listening for the signal you read, the decision you made under time pressure with incomplete information, the evidence you had for it, and what you changed afterward so the same page never fires again. This page gives the structure that makes an incident story land in four minutes, two worked narratives from GPU fleet and serving work, the follow-ups that test whether the story is real, the version that sounds heroic and fails, and what separates the senior telling from the staff telling.

Foundational

🩺 Fleet Reliability & Observability

GPU Failure Modes and XID ErrorsWhen a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.

Core

🩺 Fleet Reliability & ObservabilitySign in

DCGM and GPU TelemetryNVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou own the on-call rota for a GPU fleet. What is allowed to wake someone at 3am, and what must not?Next in this trackDesign the controller that decides whether a GPU node is healthy. What are its states, and how do you tune its false-positive rate?Next in this trackError rate on an inference fleet tripled ten minutes after a deploy. What do you do first, and what should have caught it?

FEDITOR'S NOTE

Scored on separating the trigger from the root cause, on the detection gap as a first-class section, and on action items with owners, dates and a verification rather than aspirations.

DISCUSSION · 0

No comments yet — be the first to share your approach.