Design the system that tells you every firmware version on every device in the fleet.

Firmware skew produces intermittent faults that surface layers away from their cause, so an inventory is a debugging tool rather than a compliance exercise. What to collect, how drift appears without anyone deploying anything, and the diff that turns a week-long investigation into a query.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🖧 Hardware & Cluster Build-Out

Cluster Bring-Up: Firmware, Drivers and the StackBring-up is an ordered dependency chain and skipping a step produces a symptom that points somewhere else. Firmware first, then the operating system and kernel, then the GPU driver, then the fabric manager, then the network stack, then GPUDirect, then CUDA and NCCL, then the container and scheduling layer. On Blackwell HGX systems the fabric manager reaches the NVSwitches through a bridge device and therefore depends on the InfiniBand stack being present, which is a dependency that surprises almost everyone the first time.

Foundational

🧮 Open Weights & Serving Engines

Model Onboarding: From Hugging Face to ProductionA new open-weights model lands and someone asks how long until it is serving traffic. The answer depends on a sequence that is the same every time: read the card and the config, check engine support for the exact attention and quantization combination, size it, pull the weights, bring up one replica, validate correctness against the authors' own outputs, benchmark, then roll out behind a flag. The steps that surprise people are the download, which is hours for a trillion-parameter model, and the correctness check, which almost nobody does and which catches the wrong template.

Foundational

🩺 Fleet Reliability & Observability

Alert Design and On-Call LoadAn alert exists to change what a human does, so any alert that fires without a decision attached is a false alarm regardless of whether its condition was true. GPU fleets generate a specific set of noisy signals that look serious and are not, and separating those from the ones that need a person at three in the morning is what keeps a rotation sustainable. The measure of an alerting system is the fraction of pages that led to an action.

Foundational

🗂️ Scheduling & Orchestration

Node Lifecycle: Drain, Upgrade and ReturnA node moves through a fixed cycle between provisioning and decommissioning, and most fleet operations are one lap around it: cordon so nothing new lands, drain so running work finishes or moves, act, validate, then return to the pool. The wall-clock cost of a fleet-wide change is dominated by draining rather than by the change itself, which makes the plan a scheduling document rather than a technical one.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA vendor offers 15 percent off if you take their Ethernet fabric instead of InfiniBand. Decide.Next in this trackTwo thousand GPUs are being replaced. What do you do with the old ones?Next in this trackAn entire rack stops responding at 2 a.m. Walk me through the first thirty minutes.

FEDITOR'S NOTE

Scored on the inventory serving investigations rather than compliance, on drift arising from repairs and reimages rather than deployments, and on the alert being on divergence rather than on version age.

DISCUSSION · 0

No comments yet — be the first to share your approach.