Design a training system that survives losing a node without a human in the loop. What does elasticity cost you?

A dead node at 4 a.m. costs twenty minutes if a human restarts it and under five if the system does. The four stages of an automatic recovery, the reason a lost node takes a whole pipeline replica with it, why hot spares beat resharding, and the minutes that each stage still costs even when everything works.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Elastic and Fault-Tolerant TrainingAt 16,384 GPUs something fails every few hours, so a training job is a loop of detect, isolate, replace and resume rather than one process that runs for months. The Llama 3 record of 466 interruptions in 54 days is the reference point, and the design goal is keeping effective training time above 90% by making each event cost minutes.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsAn incident on a GPU fleet is a training run that stopped, a serving endpoint burning its error budget, or a fleet-wide symptom nobody has explained yet. The response has a shape: detect, stabilize, diagnose, repair, return through the gate, write it up. The stabilizing move (drain the node, restart from checkpoint, or shift traffic) comes before the diagnosis, because a frontier run loses more per minute than any investigation is worth. This page gives the triage order, the 3am decision tree, the spare-capacity arithmetic behind drain-and-replace, and what a fleet postmortem has to contain.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Core

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackOne GPU out of 16,000 is 15% slow and the whole run is 15% slow. How do you find it, and what is usually wrong with it?Next in this trackYou see 'Watchdog caught collective operation timeout' in the logs. What happened, and how do you find which rank caused it?Next in this trackHow do you overlap communication with compute in training, and where does the overlap stop working?

FEDITOR'S NOTE

Scored on breaking recovery into detect, reschedule, reshard-or-replace, and resume, on knowing that the parallel layout constrains what can shrink, and on being honest that elasticity is minutes of bubble and a lot of code.

DISCUSSION · 0

No comments yet — be the first to share your approach.