How many spare GPUs, nodes, cables and transceivers do you hold for a 2,048-GPU fleet?

The spares number falls out of the failure rate times the repair turnaround, and the cheap items are the ones people forget. The arithmetic for GPUs, the very different arithmetic for transceivers, and the rule that decides whether to hold a part at all.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🩺 Fleet Reliability & Observability

Alert Design and On-Call LoadAn alert exists to change what a human does, so any alert that fires without a decision attached is a false alarm regardless of whether its condition was true. GPU fleets generate a specific set of noisy signals that look serious and are not, and separating those from the ones that need a person at three in the morning is what keeps a rotation sustainable. The measure of an alerting system is the fraction of pages that led to an action.

Foundational

🧮 Open Weights & Serving Engines

Model Onboarding: From Hugging Face to ProductionA new open-weights model lands and someone asks how long until it is serving traffic. The answer depends on a sequence that is the same every time: read the card and the config, check engine support for the exact attention and quantization combination, size it, pull the weights, bring up one replica, validate correctness against the authors' own outputs, benchmark, then roll out behind a flag. The steps that surprise people are the download, which is hours for a trillion-parameter model, and the correctness check, which almost nobody does and which catches the wrong template.

Foundational

🖧 Hardware & Cluster Build-Out

Colocation, Power Contracts and Site SelectionFor most organizations the constraint on deploying GPUs is not the GPUs. It is finding a hall that can deliver 100 kilowatts or more per rack, reject that heat with liquid, and sign a contract for the power years before the hardware exists. Colocation contracts price reserved capacity rather than consumption, cooling capability is what eliminates most sites, and the lead time on new electrical supply is measured in years while GPUs arrive in months.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

ECC, Row Remapping and Memory ErrorsHBM stacks flip bits, and the difference between a fleet that shrugs and one that loses a training step to corruption is error-correcting codes plus the machinery that retires bad memory before it produces a double-bit error. A single-bit error is corrected silently and counted; a double-bit error is detected, kills the process, and on Ampere and later triggers the row remapper to swap the failing row for a spare at the next reset. This page explains the codes, the remapper's states, how to read the counters as a prediction of failure, and the RMA rules a fleet applies.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackTwenty-eight of sixty-four new nodes fail acceptance. What do you do first?Next in this trackA GPU cluster has four separate networks. Name them and say what each one carries.Next in this trackDesign the fabric for 10,000 GPUs. Two tiers or three, and what does the choice cost?

FEDITOR'S NOTE

Scored on deriving the pool from failure rate times replacement time, on treating cheap high-count parts differently from expensive ones, and on the correlated-failure buffer.

DISCUSSION · 0

No comments yet — be the first to share your approach.