Design a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.

Every rank writes its own shard, a metadata file maps every chunk to a global tensor, and a loader intersects the old layout with the new. The bytes per rank for a 405B, the manifest that makes the write atomic, the metadata that makes resharding possible, and the host-memory staging that keeps the GPUs running.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Checkpoint I/OA checkpoint of a 70B model's training state is 1.13 TB, and a 405B model's is 6.5 TB. Written synchronously every half hour, it stalls thousands of GPUs for as long as the slowest rank takes to reach the disk. The design that scales writes each rank's shard in parallel, copies it off the GPU to host memory first so the run resumes in seconds, and drains it to durable storage in the background. The arithmetic is bytes per rank against the bandwidth of each hop, and the goal is a checkpoint that costs the run under 1% of its time.

Advanced

🕸️ Distributed Training🔒 Premium

Checkpointing and Resumption at ScaleA training checkpoint at frontier scale is terabytes of sharded optimizer state that must be written often enough to bound lost work and fast enough not to stall the job. The interval is a formula in the failure rate and the write cost, and asynchronous sharded writes are what turn it from a 15% tax into a 3% one.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Core

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour 4,096-GPU run loses about 2% of every day to restarts. Fix it, and tell me where the floor is.Next in this trackPlan a 10-trillion-token pre-training run end to end: compute, fleet, layout, data, checkpoints and a schedule with a failure budget.Closest to this topicDesign a training system that survives losing a node without a human in the loop. What does elasticity cost you?

FEDITOR'S NOTE

Scored on the metadata design (global shape, chunk offsets, storage location per chunk) and the intersection algorithm on load, on atomic commit via a manifest, and on the two-stage async write.

DISCUSSION · 0

No comments yet — be the first to share your approach.