Advanced

Checkpoint I/O

A checkpoint of a 70B model's training state is 1.13 TB, and a 405B model's is 6.5 TB. Written synchronously every half hour, it stalls thousands of GPUs for as long as the slowest rank takes to reach the disk. The design that scales writes each rank's shard in parallel, copies it off the GPU to host memory first so the run resumes in seconds, and drains it to durable storage in the background. The arithmetic is bytes per rank against the bandwidth of each hop, and the goal is a checkpoint that costs the run under 1% of its time.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Checkpointing and Resumption at Scale Parallel Filesystems vs Object Storage GPUDirect RDMA and GPUDirect Storage Training Uptime and Interruption Statistics Elastic and Fault-Tolerant Training NVLink, NVSwitch and PCIe

LESSONS THAT TEACH THIS

Training at ScaleCheckpoint cadence is an optimisation against your own failure rate16 min→Fleet and ReliabilityStorage is a throughput problem until the files get small14 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismDesign a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.→Napkin Math, Cost & CapacityEstimate how long it takes to write a checkpoint for a 405B training run→Hardware, Cabling & Cluster Build-OutSize the storage for a 2,048-GPU training cluster. What numbers actually drive it?→Napkin Math, Cost & CapacityWhat storage throughput does a 16k-GPU training cluster need?→Distributed Training & ParallelismAt 16,000 GPUs something fails every few hours. How do you choose the checkpoint interval, and what does the write have to look like?→Napkin Math, Cost & CapacityEstimate the activation memory for training a 70B model at an 8k sequence length→