Design the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?

Design snapshot ownership, background writes and manifest publication separately, with explicit checkpoint bytes and recovery scope.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Design snapshot ownership, background writes and manifest publication separately, with explicit checkpoint bytes and recovery scope.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🔌 Networking & Storage🔒 Premium

GPUDirect RDMA and GPUDirect StorageGPUDirect enables supported devices to transfer data to or from GPU memory without a host staging buffer. Its benefit depends on topology, software support and the transfer path.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingTrace batches from storage to GPU readiness, budget workers and prefetch memory, and resume from the input position saved with training state. An executable example shows how a read-ahead cursor can silently skip samples.

Foundational

🔌 Networking & Storage

Dataset Lifecycle: Ingest, Shard and RetainFollow a dataset from source records to verified shards, a pinned version and safe retention. Calculate token storage, distinguish caches from authoritative data, and learn why deleting a version does not necessarily free its shards.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingConsistent hashing limits key movement when a shard set changes. It can preserve cache affinity, but it does not guarantee that cached data is present or that load is balanced.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?Next in this trackWhich NCCL environment variables are worth knowing, what does each change, and which should you almost never set?Next in this trackHow do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?

FEDITOR'S NOTE

Separate the training pause from recoverable publication. Count unique writer payloads, protect snapshot buffers and show whether background writes keep up.

DISCUSSION · 0

No comments yet — be the first to share your approach.