← AI Infrastructure System Design / 26

hardAnthropicMeta

Design disaster recovery for a three-month training run: checkpoint replication, cluster failover, and the RTO you can promise.

Derive checkpoint replication and recovery budgets, identify the state a second cluster needs, and test the difference between a copied artifact and resumed training.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Derive checkpoint replication and recovery budgets, identify the state a second cluster needs, and test the difference between a copied artifact and resumed training.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems Design

Multi-Region Serving and FailoverMulti-region serving can improve geographic reach and survive a regional outage. It requires enough remaining capacity and a clear plan for state, routing and degraded service.

🔌 Networking & StorageSign in

Parallel Filesystems vs Object StorageChoose storage by access semantics, working-set reuse and failure protection. Trace a shared read/checkpoint budget, distinguish local completion from durable export, and learn when a second storage tier earns its cost.

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainConvert a stated work budget into elapsed time and GPU-hours. Vary utilization, invert for a deadline, and separate productive time from calendar delays without counting them twice.

🕸️ Distributed Training

Data Parallelism and DDPReplicas combine gradients from different data. Follow a two-worker update, fix unequal-token weighting, and see why queued gradient buckets can leave more than one reduction exposed.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign token-level billing and metering for an LLM API: count at the engine, aggregate, reconcile, and survive the failure modes.Next in this trackDesign prefill and decode disaggregation for a 70B serving fleet: pool sizes, KV transfer, routing, and when to turn it on.Next in this trackDesign an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.

FEDITOR'S NOTE

Assess explicit RPO and RTO assumptions, complete remote checkpoints, feasible recovery capacity and correctness at the intended scale.

DISCUSSION · 0

No comments yet — be the first to share your approach.