Your new rack is tripping the contracted power limit during training. What do you do?

A training cluster is a synchronized load, so the peak is far above the average and the average is what everyone budgeted. The immediate mitigation takes minutes and costs a few percent of throughput, and the four durable fixes trade cost against throughput in different places.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🖧 Hardware & Cluster Build-Out

Rack Power Delivery and BuswaysA GPU rack has gone from 10 kW to over 120 kW in a few generations, and the electrical design changed with it. At 132 kW on a 415 V three-phase feed a rack draws about 184 amps, which is past what a normal power strip carries, so distribution moves to overhead busway and the rack takes redundant high-current taps. On top of the steady draw sits a synchronized transient every training step, because thousands of GPUs finish a collective at the same instant, and that swing is what sizes the upstream equipment.

Foundational

🖧 Hardware & Cluster Build-Out

Cluster Bring-Up: Firmware, Drivers and the StackBring-up is an ordered dependency chain and skipping a step produces a symptom that points somewhere else. Firmware first, then the operating system and kernel, then the GPU driver, then the fabric manager, then the network stack, then GPUDirect, then CUDA and NCCL, then the container and scheduling layer. On Blackwell HGX systems the fabric manager reaches the NVSwitches through a bridge device and therefore depends on the InfiniBand stack being present, which is a dependency that surprises almost everyone the first time.

Foundational

🧮 Open Weights & Serving Engines

Model Onboarding: From Hugging Face to ProductionA new open-weights model lands and someone asks how long until it is serving traffic. The answer depends on a sequence that is the same every time: read the card and the config, check engine support for the exact attention and quantization combination, size it, pull the weights, bring up one replica, validate correctness against the authors' own outputs, benchmark, then roll out behind a flag. The steps that surprise people are the download, which is hours for a trillion-parameter model, and the correctness check, which almost nobody does and which catches the wrong template.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Thermal, Power and Cooling EventsA GPU that gets too hot or is denied power does not fail; it slows down, and on a synchronous job a slow GPU is a slow job. Thermal and power events are the most common cause of the 'nothing failed but the run is 15% slower' ticket, and they are the incidents that scale from one node to a whole hall when a cooling distribution unit or a power feed has a problem. This page explains how throttling works, derives the step-time cost of a clock reduction, walks the failure modes of air and liquid cooling, and covers the power behaviour peculiar to training: thousands of GPUs going idle and busy in lockstep.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackCount the switches, cables and transceivers for a 2,048-GPU rail-optimized fabric.Next in this trackAfter a maintenance window, two-node bandwidth halved. Single-node is unchanged. Diagnose it.Next in this trackSize the storage for a 2,048-GPU training cluster. What numbers actually drive it?

FEDITOR'S NOTE

Scored on identifying the synchronized transient rather than a sizing error, on power capping as the immediate lever with its measured cost, and on the four durable options with their trades.

DISCUSSION · 0

No comments yet — be the first to share your approach.