mediumNewMetaMicrosoft

Estimate how long it takes to write a checkpoint for a 405B training run

Sixteen bytes per parameter is 6.5 TB per checkpoint; over a 100 GB/s filesystem that is about a minute, which at a 30-minute cadence is 3% of the run. The chain, the storage tiers that meet it, and why the write is made asynchronous.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Checkpoint I/OA checkpoint of a 70B model's training state is 1.13 TB, and a 405B model's is 6.5 TB. Written synchronously every half hour, it stalls thousands of GPUs for as long as the slowest rank takes to reach the disk. The design that scales writes each rank's shard in parallel, copies it off the GPU to host memory first so the run resumes in seconds, and drains it to durable storage in the background. The arithmetic is bytes per rank against the bandwidth of each hop, and the goal is a checkpoint that costs the run under 1% of its time.

Core

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainThe fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.

Foundational

🩺 Fleet Reliability & Observability

Alert Design and On-Call LoadAn alert exists to change what a human does, so any alert that fires without a decision attached is a false alarm regardless of whether its condition was true. GPU fleets generate a specific set of noisy signals that look serious and are not, and separating those from the ones that need a person at three in the morning is what keeps a rotation sustainable. The measure of an alerting system is the fraction of pages that led to an action.

Foundational

🧭 Ownership & Judgment

The Reliability Pushback StoryEvery AI infra loop has a behavioral round, and the story it wants most is the one where you stopped something (a launch, a run, a hardware admission) because the data said to, and you were accountable for the cost of stopping. This page gives the skeleton that works: the situation, the signal you read, the decision and who owned it, the evidence you brought, and what changed afterward. It also gives the follow-up interviewers hold back, the version that sounds right and fails, and the line between a senior telling and a staff telling of the same story.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat is the power budget for a cluster of 1,024 H100s?Next in this trackShould we buy or rent 1,000 H100s for the next two years?Next in this trackHow much does moving from bf16 to fp8 save in serving cost?

FEDITOR'S NOTE

The interviewer wants the candidate to know what is in a checkpoint (optimizer state dominates), to size it, and to convert the write time into lost MFU at a given cadence. The async answer is the follow-up.

DISCUSSION · 0

No comments yet — be the first to share your approach.