Coolant flow to a rack stops. What happens in the next sixty seconds, and what has to be automatic because a human cannot act in time?

A cold plate holds a few tens of grams of water against seven hundred watts, so temperature rises about two degrees a second and the limit arrives in under twenty. What that timescale rules out, the protection layers that act inside it, and the choice that decides whether you lose a job or the hardware.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Thermal, Power and Cooling EventsA GPU that gets too hot or is denied power does not fail; it slows down, and on a synchronous job a slow GPU is a slow job. Thermal and power events are the most common cause of the 'nothing failed but the run is 15% slower' ticket, and they are the incidents that scale from one node to a whole hall when a cooling distribution unit or a power feed has a problem. This page explains how throttling works, derives the step-time cost of a clock reduction, walks the failure modes of air and liquid cooling, and covers the power behaviour peculiar to training: thousands of GPUs going idle and busy in lockstep.

Foundational

🖧 Hardware & Cluster Build-Out

Direct-to-Chip Liquid Cooling and CDUsAbove roughly 40 kW a rack cannot be cooled by air in any practical hall, which is why every dense GPU deployment now runs liquid to the chip. A cold plate sits on each GPU, a coolant distribution unit isolates the clean rack loop from facility water, and the facility side runs warm, typically 30 to 40 degrees supply, because warm water is cheaper to make. The design numbers are flow rate and temperature rise, and both fall out of one equation that every operator should be able to do from memory.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Power and Datacenter ConstraintsThe binding constraint on new GPU capacity in 2026 is not chips or capital but megawatts: an H100 node draws about 10 kW, a GB200 NVL72 rack about 120 kW, and a 100,000-GPU cluster needs on the order of 150 MW with cooling. This page converts GPU counts to power, power to cooling and facility requirements, and both to cost, so a candidate can size a training hall from a power budget and explain why liquid cooling, PUE and the local grid decide where the next cluster goes.

Foundational

🩺 Fleet Reliability & Observability

GPU Failure Modes and XID ErrorsWhen a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackTrace one request through an inference platform. What are the spans, what does each cost, and where does latency actually hide?Next in this trackOne serving replica has a per-token latency 40 percent worse than its peers. Find out why.Next in this trackWrite the policy for when a GPU is replaced rather than returned to service. What are the triggers and what do they cost?

FEDITOR'S NOTE

Scored on the thermal-mass arithmetic giving tens of seconds, on the conclusion that every protection layer must be automatic, and on the ordering of throttle, checkpoint attempt and shutdown.

DISCUSSION · 0

No comments yet — be the first to share your approach.