Design the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?

Six and a half terabytes leaving sixteen thousand GPUs at once is a burst no shared filesystem absorbs, and the fix is to stop trying. What the barrier actually has to wait for, the per-node arithmetic that makes it seconds, and the manifest rule that decides whether a checkpoint is usable at all.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

GPUDirect RDMA and GPUDirect StorageBy default a byte leaving a GPU for the network or the disk makes a detour through host memory, crossing PCIe twice and costing a CPU copy. GPUDirect RDMA lets the NIC read and write GPU memory directly, and GPUDirect Storage does the same for NVMe. The win is not raw bandwidth (PCIe is the ceiling either way) but the halving of PCIe traffic and the removal of the host as a bottleneck, which is what makes collectives run at NIC rate and checkpoints run at drive rate. When it is silently off, everything still works, at half speed.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingThe dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.

Foundational

🔌 Networking & Storage

Dataset Lifecycle: Ingest, Shard and RetainA training dataset is not a file, it is a pipeline with four stages and a retention policy, and each stage has a different bottleneck. Ingest is metadata-bound rather than bandwidth-bound. Tokenization is CPU work that should happen once offline rather than every epoch. Sharding decides whether the training read is a stream or a storm of small files. And retention decides how much of the bill is paid for bytes nobody reads.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingSplitting work across N servers with a modulo of N moves almost everything when N changes, which for a cache means throwing away almost all of it. Consistent hashing places servers and keys on a ring so adding or removing one moves only its share, and virtual nodes fix the imbalance a small ring otherwise has. In LLM serving the same structure routes requests by prompt prefix so a conversation reaches the replica already holding its cache.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?Next in this trackWhich NCCL environment variables are worth knowing, what does each change, and which should you almost never set?Next in this trackHow do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?

FEDITOR'S NOTE

Scored on the burst arithmetic per node rather than in aggregate, on the barrier waiting only for the host copy, and on manifest-last as the property that makes a checkpoint either complete or absent.

DISCUSSION · 0

No comments yet — be the first to share your approach.