Your 4,096-GPU run loses about 2% of every day to restarts. Fix it, and tell me where the floor is.

Turn the daily loss into recovery and replay costs, then identify which part is measured and which is assumed. Compare checkpoint cadence, recovery work and failure prevention without promising one universal floor.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

GPUs and time-to-train calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🕸️ Distributed Training🔒 Premium

Elastic and Fault-Tolerant TrainingRecover the intended training run after a failure: fence old workers, restore a supported layout, reject stale checkpoint uploads and measure progress that survives. Includes a generation-check example and replay budget.

Advanced

🕸️ Distributed Training🔒 Premium

Checkpointing and Resumption at ScaleLearn what a training checkpoint must save, why an asynchronous snapshot needs isolation, and when a complete write becomes a safe restore point. Work through a 56 GB state budget and test the continuation.

Foundational

🩺 Fleet Reliability & Observability

Training Uptime and Interruption StatisticsTraining interruption statistics connect failure frequency to lost progress. Use a stated rate, checkpoint cost and recovery model to estimate how much time a run can lose.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Incident Response for GPU FleetsGPU incident response restores useful service while preserving enough evidence to diagnose the failure. Mitigation, repair and return-to-service checks have different purposes.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackPlan a 10-trillion-token pre-training run end to end: compute, fleet, layout, data, checkpoints and a schedule with a failure budget.Next in this trackDesign a distillation pipeline that improves a target task without hiding capability regressions Next in this trackA PyTorch team wants to train on TPUs. Do they rewrite in JAX with MaxText, use PyTorch/XLA, or wait for TorchTPU?

FEDITOR'S NOTE

Define useful progress and the denominator behind the reported loss. Separate event rate, recovery, replay and checkpoint costs; optimize the measured terms while preserving safe failure detection and recoverable state.

DISCUSSION · 0

No comments yet — be the first to share your approach.