Design a checkpoint format for thousands of GPUs: no gather on write, resumable at a different world size, no stall.

Describe logical tensors independently of writer ranks, publish only a complete checkpoint, and reconstruct the target layout from stored chunks. Asynchronous saving reduces blocking but still needs a consistent snapshot and a memory budget.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.