A GPU in a running 512-GPU training job is throwing errors. How do you get it out of the job without losing the run?

Keep the training state and replace the failed capacity. Trace fault detection, safe checkpoint selection, worker shutdown and restart; size spare capacity from observed failures and repair times.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Keep the training state and replace the failed capacity. Trace fault detection, safe checkpoint selection, worker shutdown and restart; size spare capacity from observed failures and repair times.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🕸️ Distributed Training🔒 Premium

Elastic and Fault-Tolerant TrainingRecover the intended training run after a failure: fence old workers, restore a supported layout, reject stale checkpoint uploads and measure progress that survives. Includes a generation-check example and replay budget.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsGPU incident response restores useful service while preserving enough evidence to diagnose the failure. Mitigation, repair and return-to-service checks have different purposes.

Advanced

🔌 Networking & Storage🔒 Premium

Checkpoint I/OTrace checkpoint bytes through staging and storage, calculate when background saves fall behind, and distinguish upload completion from recoverable-state age. Includes an executable queue example and restore budget.

Foundational

🗂️ Scheduling & Orchestration

Node Lifecycle: Drain, Upgrade and ReturnFleet maintenance moves nodes through drain, repair and validation before returning them to service. Deadlines and concurrency limits keep that work within the available capacity.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackExplain how a Slurm job actually gets its GPUs: GRES, partitions, cgroups, and the sbatch that lands 8 GPUs on one node.Next in this trackWe want to run LLM inference on Kubernetes. What does the deployment actually need beyond a Deployment with a GPU limit?Next in this trackDesign the scheduling and isolation for a multi-tenant fine-tuning service: hundreds of customers, a few base models, shared GPUs.

FEDITOR'S NOTE

Distinguish application recovery from hardware repair. Account for checkpoint age and restart time separately, and do not assume elastic training preserves live workers or eliminates recovery.

DISCUSSION · 0

No comments yet — be the first to share your approach.