How would you build a postmortem practice that people take seriously and that actually reduces incidents?

Most postmortem processes fail in one of three predictable ways, and each has a mechanical fix rather than a cultural exhortation. The threshold that decides which incidents get one, the review that changes the document, the tracking that closes the items, and the measurement that says whether any of it is working.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsAn incident on a GPU fleet is a training run that stopped, a serving endpoint burning its error budget, or a fleet-wide symptom nobody has explained yet. The response has a shape: detect, stabilize, diagnose, repair, return through the gate, write it up. The stabilizing move (drain the node, restart from checkpoint, or shift traffic) comes before the diagnosis, because a frontier run loses more per minute than any investigation is worth. This page gives the triage order, the 3am decision tree, the spare-capacity arithmetic behind drain-and-replace, and what a fleet postmortem has to contain.

Core

🧭 Ownership & JudgmentSign in

On-Call Narratives That LandEvery infrastructure loop has a round where you are asked to tell an incident story, and the interviewer is not listening for drama. They are listening for the signal you read, the decision you made under time pressure with incomplete information, the evidence you had for it, and what you changed afterward so the same page never fires again. This page gives the structure that makes an incident story land in four minutes, two worked narratives from GPU fleet and serving work, the follow-ups that test whether the story is real, the version that sounds heroic and fails, and what separates the senior telling from the staff telling.

Foundational

🩺 Fleet Reliability & Observability

GPU Failure Modes and XID ErrorsWhen a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.

Core

🩺 Fleet Reliability & ObservabilitySign in

DCGM and GPU TelemetryNVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour fleet is replacing one percent of its GPUs every month. Is that normal, and how would you find out?Closest to this topicWrite the postmortem for a training run that lost twelve hours. What goes in it, and what makes the action items stick?Closest to this topicHow do you run a postmortem that actually changes something?

FEDITOR'S NOTE

Scored on mechanical fixes rather than appeals to culture, on a written threshold for which incidents qualify, on tracking action items to closure, and on measuring recurrence as the outcome.

DISCUSSION · 0

No comments yet — be the first to share your approach.