How much spare capacity does a 16,000-GPU fleet need, and what are you actually reserving it for?

Three separate reserves get merged into one number and then argued about. The repair pipeline from Little's law, the restart pool that has to be instantly available, and the correlated-failure buffer sized by the largest thing that can fail at once, each derived and then added.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Capacity Planning and UtilizationCapacity planning for GPUs is deciding how many to have next quarter given that they cost money whether busy or not, that demand arrives in bursts, and that a queue near saturation produces waits that grow without bound. This page works the planning arithmetic for a serving fleet (peak demand, headroom, the p99 penalty of running hot) and a training platform (job mix, queue wait, the value of a shared pool), and gives the queueing intuition that makes 70% look full. The number that decides everything is utilization, and it has a ceiling set by latency, not by hardware.

Foundational

🩺 Fleet Reliability & Observability

GPU Failure Modes and XID ErrorsWhen a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.

Core

🩺 Fleet Reliability & ObservabilitySign in

DCGM and GPU TelemetryNVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

ECC, Row Remapping and Memory ErrorsHBM stacks flip bits, and the difference between a fleet that shrugs and one that loses a training step to corruption is error-correcting codes plus the machinery that retires bad memory before it produces a double-bit error. A single-bit error is corrected silently and counted; a double-bit error is detected, kills the process, and on Ampere and later triggers the row remapper to swap the failing row for a spare at the next reset. This page explains the codes, the remapper's states, how to read the counters as a prediction of failure, and the RMA rules a fleet applies.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow would you build a postmortem practice that people take seriously and that actually reduces incidents?Next in this trackYour fleet is replacing one percent of its GPUs every month. Is that normal, and how would you find out?Closest to this topicWrite the policy for when a GPU is replaced rather than returned to service. What are the triggers and what do they cost?

FEDITOR'S NOTE

Scored on separating the three reserves rather than quoting a single percentage, on Little's law for the repair pipeline, and on the largest correlated failure domain sizing the third.

DISCUSSION · 0

No comments yet — be the first to share your approach.