TL;DR: Use the job's interruption rate and measured checkpoint cost. A first-order model balances checkpoint overhead C/T against expected lost work T/(2M), giving T≈√(2CM). With M=180 minutes and C=1 minute, that is about 19 minutes of useful work between checkpoints. It is an approximation, not an exact availability guarantee or a universal rate for a 16k-GPU fleet.
How to approach it
Define an interruption that actually restarts this job, the checkpoint's recoverability boundary, exposed save cost and recovery time. Derive the tradeoff, check its assumptions, then test the policy with real restore events.
A strong answer
MTBF is mean time between failures; here M means mean time between job interruptions. A large training job is interrupted about once every three hours in its measured history. The next decision is how much progress to risk before saving again. GPU count alone does not supply that interruption rate: hardware, networking, maintenance and failure containment all contribute.
Use illustrative model inputs M=180 minutes between job interruptions, C=1 minute of exposed checkpoint cost and R=5 minutes to recover. Let T be useful computation between checkpoints. When overheads are small relative to the relevant intervals, a first-order estimate is:
overhead(T) ≈ C/T + T/(2M) + R/M
saves + lost work + recovery
stationary point: −C/T² + 1/(2M) = 0
T* = √(2CM) = √360 = 18.9737 minutes
The T/2 loss approximation treats failures as roughly uniform across the useful-work interval; independent constant-rate failures and T small relative to M motivate it. Exact renewal models must also account for failures during checkpointing/recovery and the time spent on those activities. Checkpointing research discusses interval selection and fault-prediction assumptions. The formula here is deliberately a first-order budgeting model.
At the stationary point, the two variable costs match:
checkpoint term = 1/18.9737 = 5.2705%
lost-work term = 18.9737/360 = 5.2705%
recovery term = 5/180 = 2.7778%
total modeled overhead ≈ 13.3187%
remaining useful-time fraction ≈ 86.6813%
sanity: save and lost-work terms are equal at the model optimum.
That last fraction is not a measured throughput result. If C rises to five minutes with M and R unchanged, the modeled best interval becomes √1800=42.4264 minutes. The save and lost-work terms each become 11.7851%; the useful-time estimate falls to 73.6520%, about 13.03 percentage points lower. Recheck the approximation as checkpoint cost becomes a material fraction of the interval.
| Change, other inputs fixed | First-order consequence |
|---|---|
| C doubles | Interval multiplies by √2 |
| Interruption rate doubles | Interval divides by √2 |
| R increases | More overhead; same model optimum |
| Failures become correlated | Refit the interruption model |
An asynchronous checkpoint can return quickly while its data is still being written. A copy in host memory may survive a process failure but disappear if the host fails. Measure when the checkpoint becomes recoverable for each failure you intend to handle. Include exposed stalls and background I/O slowdown in the cost, and track the age of the last usable checkpoint. A slow write can put much more progress at risk than the nominal save interval suggests.
If every one of N independent, identical components interrupted the entire job at rate λ, the aggregate rate would be Nλ. Those assumptions often fail: a shared switch failure is one correlated event, while a corrected device event might not interrupt the job at all. Training Uptime and Interruption Statistics provides the event-counting context; Checkpointing and Resumption at Scale covers the restore contract. Estimate from job-level events and record their causes. Do not promote one fleet's observed component rate into a universal constant.
Start near the model's interval, then compare completed, recoverable progress per wall hour under representative failures. Reverse the policy if checkpoint I/O creates long tails, restore tests fail, or the observed lost-work distribution disagrees with the model. A successful write call is not a successful restart test.
What interviewers probe next
- “Why not save every minute?” At C=1 minute, C/T is already 100% in this first-order expression; the small-overhead approximation has broken down.
- “Does faster recovery change T?”* R is a constant term in this simplified model. A richer model can couple recovery and failure behavior differently.
- “What belongs in the checkpoint?” Whatever is required for the promised restart semantics: model/optimizer state and relevant progress/random/data state, with compatible restore code.
Common mistakes
- Treating all GPU faults as independent job interruptions.
- Counting a non-durable staging copy as protected progress.
- Calling a first-order useful-time estimate measured goodput.
Key takeaways
- M=180 and C=1 give a modeled interval of 18.97 minutes.
- The variable overhead terms balance at about 5.27% each.
- Use the age of the last recoverable checkpoint when measuring lost work.