AI Infra Interviews logo
GPU Fleet Reliability & Observability / 18
mediumNewBasetenOpenAI

Error rate on an inference fleet tripled ten minutes after a deploy. What do you do first, and what should have caught it?

Roll back first and investigate afterward, because the error budget is being consumed while you read logs. The rollback decision rule that removes the argument, the four causes specific to model serving, and the canary design that would have caught it at one percent of the traffic.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Roll back first and investigate afterward, because the error budget is being consumed while you read logs. The rollback decision rule that removes the argument, the four causes specific to model serving, and the canary design that would have caught it at one percent of the traffic.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core
🩺 Fleet Reliability & ObservabilitySign in
SLOs for AI SystemsA service level objective is a promise with a number attached, and AI systems need their own because the classic ones do not fit: a training run has no requests, only progress, so its objective is goodput; an LLM endpoint streams, so its latency is two numbers (time to first token and time per token) rather than one; and both spend a budget that is set by hardware failure rates rather than by software bugs. This page defines the objectives that fleet and serving teams actually use, derives the thresholds from user needs and from the hardware, and works the error-budget arithmetic that decides when to stop shipping and start fixing.
Advanced
🧭 Ownership & Judgment🔒 Premium
Migrations and DeprecationsEvery infrastructure career contains a migration nobody wanted: the scheduler swap, the driver upgrade across a live fleet, the storage move while training runs are in flight, the deprecation of the launcher every team's scripts depend on. The behavioral round asks about one because it tests the skills that matter most and show least on a résumé: sequencing under risk, keeping a rollback real, moving people who have no reason to move, and knowing when to stop. This page gives the shape of a migration story that lands, two worked narratives from GPU fleet work, and the answers that sound like leadership and fail.
Foundational
🩺 Fleet Reliability & Observability
GPU Failure Modes and XID ErrorsWhen a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.
Core
🩺 Fleet Reliability & ObservabilitySign in
DCGM and GPU TelemetryNVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.
UP NEXT ON YOUR JOURNEY
FEDITOR'S NOTE

Scored on rolling back before diagnosing, on the specific serving failure modes rather than generic deploy bugs, and on a canary with an automatic promotion and rollback rule.

DISCUSSION · 0

No comments yet — be the first to share your approach.