TL;DR: Three numbers decide the cadence: the mean time between interruptions, which at 16,384 GPUs is about 3.1 hours from the published failure rate; the cost of taking a checkpoint, which with a local-first write path is about a minute; and the restart time, about 5 minutes. The optimal interval is the square root of twice the checkpoint cost times the mean time between failures, which for those numbers is about 19 minutes. Goodput is then one minus three losses: the checkpoint overhead, which is cost over interval; the work lost per failure, which averages half an interval over the mean time between failures; and the restart time over the mean time between failures. At the optimum those come to 5.2%, 5.2% and 2.7%, leaving about 87% goodput, which is close to what large published runs report. The important consequence is that the two overhead terms are equal at the optimum, so the way to improve goodput is not to argue about the interval but to make checkpoints cheaper: at a 5-minute checkpoint cost the same arithmetic gives only 74%.
How to approach it
Get the three inputs first and say where each comes from, since two are measured and one is derived from fleet size. State the formula and show it balances two competing costs. Compute goodput as a sum of three named losses so the breakdown is legible. Then vary the checkpoint cost, because that comparison is the actionable finding and the interval by itself is not.
A strong answer
A typical situation: a team debates whether to checkpoint every 15 or every 45 minutes, with strong opinions on both sides. Both intervals are within a few percent of each other in goodput. Their checkpoint takes 5 minutes because it writes synchronously to shared storage, and fixing that is worth four times more than either choice.
The three inputs:
mean time between interruptions, M
from the published rate of about 2e-5 interruptions per GPU-hour
at 16,384 GPUs: 2e-5 x 16,384 = 0.32 per hour, so M = 1 / 0.32 = 3.1 hours = 186 minutes
this is derived from fleet size and should be replaced by your own measured rate
checkpoint cost, C
the time the job is not making progress because it is checkpointing
with a local-first write path where the barrier waits only for a copy into host memory:
about 1 minute including the barrier and the quiesce
with a synchronous write to shared storage: 5 minutes or more
restart time, R
detection, rescheduling, process start, weight load, and reaching the first step
about 5 minutes on a well-automated platform
The formula and where it comes from:
two costs move in opposite directions as the interval T changes:
checkpoint overhead = C / T falls as T grows
expected work lost = T / (2M) rises as T grows, since a failure lands on average
halfway through an interval
total overhead f(T) = C/T + T/(2M)
minimize: df/dT = -C/T^2 + 1/(2M) = 0 -> T^2 = 2CM -> T = sqrt(2 C M)
with C = 1 min and M = 186 min:
T_opt = sqrt(2 x 1 x 186) = sqrt(372) = 19.3 minutes
sanity: at the optimum the two terms are equal by construction, C/T = T/(2M), which is a
useful check on any answer: if a proposed cadence makes one term much larger than the
other, it is not optimal
Checkpointing and Resumption at Scale has the derivation in full; Training Uptime and Interruption Statistics is the source of the rate.
The goodput that leaves:
goodput = 1 - (checkpoint overhead) - (lost work) - (restart cost)
= 1 - C/T - T/(2M) - R/M
at C = 1, T = 19.3, M = 186, R = 5:
checkpoint overhead = 1 / 19.3 = 5.2%
lost work = 19.3 / 372 = 5.2%
restart = 5 / 186 = 2.7%
goodput = 1 - 0.131 = 86.9%
sanity: large published runs report effective training time near 90%, so 87% from these
inputs is the right order and the small gap is planned maintenance and stragglers,
which this formula does not model
What actually moves the number:
| Checkpoint cost C | Optimal interval | Checkpoint overhead | Lost work | Restart | Goodput |
|---|---|---|---|---|---|
| 1 minute | 19.3 min | 5.2% | 5.2% | 2.7% | 86.9% |
| 2 minutes | 27.3 min | 7.3% | 7.3% | 2.7% | 82.6% |
| 5 minutes | 43.1 min | 11.6% | 11.6% | 2.7% | 74.1% |
reading the table: going from a 5-minute checkpoint to a 1-minute one is 12.8 points of
goodput. On a 16,384-GPU fleet at $2.5 per GPU-hour over a 720-hour month that is
0.128 × 16,384 × 2.5 × 720 = $3.8M a month of recovered capacity. Call it capacity, not
cash: it is only a saving if you would otherwise have rented more, and only revenue if the
freed hours get sold
choosing the interval badly costs far less: at C = 1 minute, using 40 minutes instead of the
optimal 19 gives 1/40 + 40/372 + 2.7% = 2.5% + 10.8% + 2.7% = 84.0%, only 2.9 points worse
sanity: the interval is a shallow optimum and the checkpoint cost is not, which is why the
engineering effort belongs in the write path rather than in the scheduling debate
Checkpoint I/O covers how to get C down to a minute, which is the local-first write path with an asynchronous drain.
Measuring the two inputs rather than assuming them takes no new instrumentation. C is the GPU idle time at checkpoint steps, which the per-step timing already records: take the median gap at steps where a checkpoint fired. M comes from the job lifecycle events, counting unexpected terminations over run hours. Recompute the optimum monthly, because both drift: C changes when the storage path changes, and M changes as the fleet ages or grows.
The reversal condition: this model assumes failures are independent and uniformly distributed in time, which is the usual approximation and is wrong in two specific ways. Correlated failures, such as a power event taking a rack or a bad firmware rollout, arrive together and defeat the assumption; the response is not a different cadence but a different failure domain, since no checkpoint interval helps if the checkpoint's storage went down too. And the rate is not constant over a run: it is higher early, while marginal hardware is being weeded out, which argues for a shorter interval in the first days and relaxing it once the fleet has settled. Both are refinements on a model that is right to within a few points, which is the accuracy the decision needs.
What interviewers probe next
- "What if the checkpoint is asynchronous?" Then C is only the barrier and copy time, not the write, which is exactly why the local-first path is the highest-value change: it moves C from minutes to about a minute or less.
- "Does the formula change with fleet size?" Only through M, which is inversely proportional to GPU count. Doubling the fleet halves M and shortens the optimal interval by the square root of two.
- "How would you validate the model?" Compare predicted goodput against measured over a month. A persistent gap means an unmodeled loss, usually stragglers or maintenance, and the breakdown says which.
- "What about elastic training?" It changes R rather than the formula: continuing at reduced size without a full restart cuts the restart term and slightly reduces the lost work, since the job never fully stops.
Common mistakes
- Choosing a cadence by convention rather than computing it, when the formula needs three numbers you already have.
- Optimizing the interval while the checkpoint takes five minutes, which is where the goodput actually is.
- Using a mean time between failures for one GPU rather than for the fleet.
- Forgetting the restart term, which is 2.7% here and is often the easiest of the three to reduce.
Key takeaways
- Optimal interval is the square root of 2 times checkpoint cost times mean time between failures: about 19 minutes at C of 1 minute and M of 186.
- At the optimum the checkpoint overhead and the lost work are equal, which is a check on any proposed cadence.
- Goodput = 1 minus C/T minus T/(2M) minus R/M, giving 87% for these inputs, matching published runs.
- Reducing checkpoint cost from 5 minutes to 1 is worth 12.8 points of goodput; choosing the interval badly costs about 3.