Training Uptime and Interruption Statistics

TL;DR: A large job has many opportunities for interruption, so a low per-device rate can still matter at fleet scale. Estimate replayed work and recovery time under explicit assumptions. Historical data is a useful reference, but a new fleet needs its own measurements, especially for correlated failures and software incidents.

The dataset

The Llama 3 paper provides a detailed public interruption dataset that is useful for a worked example.

Two things stand out. Failures are dominated by the GPU and its memory rather than by the network, which surprises people who expect the fabric to be the weak point. And only three of 419 required significant manual intervention, meaning the run survived on automation: detection, checkpoint restore, and restart on a replacement node.

The derivation, step by step

Step one, the fleet failure interval.

unexpected stops per day = 419 ÷ 54 ≈ 7.76
mean time between stops, fleet-wide: M = 24 ÷ 7.76 ≈ 3.1 h ≈ 186 min

Step two, the per-device rate, so the number transfers to a different fleet.

GPU-hours in the window = 16,384 × 54 × 24 ≈ 21.2 million
failures per GPU-hour ≈ 419 ÷ 21.2M ≈ 2.0 × 10⁻⁵   (every stop charged to a GPU, an attribution convention for this example, not a measured physical-device failure rate)
per-GPU MTBF ≈ 16,384 × 54 × 24 ÷ 419 ≈ 50,700 h ≈ 5.8 years
annualized failure rate per GPU ≈ 8,760 ÷ 50,700 ≈ 17% per year (with the same caveat)

Step three, transfer to another fleet size.

M(N) ≈ 50,700 h ÷ N
  N = 512:      ≈ 99 h  (four days)
  N = 4,096:    ≈ 12 h
  N = 16,384:   ≈ 3.1 h
  N = 100,000:  ≈ 30 min

Step four, the cost of each stop. A checkpoint every T minutes loses on average T/2 minutes of progress when the stop lands at a random point in the interval, plus the restart cost R (detect the failure, drain, reschedule on a spare, load the checkpoint, rebuild the collective, warm up).

loss per stop ≈ T/2 + R
lost fraction ≈ (T/2 + R) ÷ M
effective training time ≈ 1 − lost fraction − checkpoint overhead, where checkpoint overhead ≈ C ÷ T
    (C = the time the run is stalled while a checkpoint is taken; near zero with async checkpoints)

Step five, put Llama 3's numbers in.

M = 186 min
T = 30 min, R = 10 min, C = 1 min:   loss = (15 + 10) ÷ 186 ≈ 13.4%;  overhead = 1 ÷ 30 ≈ 3.3%  → ~83% effective
T = 15 min, R = 5 min,  C = 1 min:   loss = (7.5 + 5) ÷ 186 ≈ 6.7%;   overhead ≈ 6.7%           → ~87% effective
T = 15 min, R = 5 min,  C ≈ 0 (async): loss ≈ 6.7%;                  overhead ≈ 0               → ~93% effective
sanity: reported effective training time was above 90%, which is consistent with a checkpoint interval
        in the tens of minutes, a restart in single-digit minutes, and checkpoints that barely stall the run.

The optimal checkpoint interval

There is a closed form for the T that minimizes the sum of the two losses (checkpoint overhead C/T, which falls as T grows, and expected rework T/2 ÷ M, which rises). Setting the derivative to zero gives the Young-Daly approximation:

T_opt ≈ √(2 × C × M)
  C = 1 min,  M = 186 min:  T_opt ≈ √(372) ≈ 19 min
  C = 5 min,  M = 186 min:  T_opt ≈ √(1,860) ≈ 43 min
  C = 1 min,  M = 30 min (100k GPUs):  T_opt ≈ √60 ≈ 8 min
  C = 0.1 min (async), M = 30 min:     T_opt ≈ √6 ≈ 2.4 min

Read the rows in pairs: cutting the checkpoint cost with asynchronous writes lets the interval shrink, and a shorter interval is what keeps the lost fraction small as the fleet grows. At 100,000 GPUs a synchronous 5-minute checkpoint is untenable (T_opt ≈ 17 min, and each stop loses 8 to 9 minutes plus restart, against a 30-minute M, so the run spends over half its time not learning); an asynchronous checkpoint every couple of minutes keeps it above 80% if the restart cost R stays under about 3.5 minutes. That single derivation is why checkpoint I/O design and fault-tolerant training get their own pages.

What the restart cost is made of

R is the term platform teams can attack after checkpointing, and it is a sum: detection (a hung collective can take the configured process-group timeout, to be noticed, which is why straggler and hang detection is its own discipline), drain and reschedule (seconds with a hot spare, minutes if a node must be found), checkpoint load (a 405B model's state is several terabytes across the ranks; from a parallel filesystem at hundreds of GB/s aggregate that is tens of seconds, from object storage minutes), collective rebuild (NCCL init across thousands of ranks, a minute or two), and warm-up. Llama 3's three incidents requiring significant manual intervention, against 416 handled without it, show the goal: R is a property of the automation, not of the on-call.

Working it in the room

Derive the observed interruption interval and state which events it includes. Use checkpoint cadence and recovery time to estimate lost progress, then include the overhead of saving checkpoints. Explain where independence and uniform failure timing simplify the model, and how measured fleet behavior could change the plan.

What to remember

• Llama 3: 16,384 H100s, 54 days, 419 unexpected stops (78% hardware, GPUs and HBM first), 3 requiring significant manual intervention, over 90% effective training time.
• Fleet interval M ≈ 50,700 GPU-hours ÷ N: 3.1 h at 16k GPUs, 30 min at 100k.
• Loss per stop ≈ T/2 + R; effective time ≈ 1 − (T/2 + R)/M − C/T.
• Young-Daly: T_opt ≈ √(2CM). Cheaper checkpoints allow shorter intervals, which is what keeps big fleets productive.
• R is detection plus reschedule plus checkpoint load plus collective rebuild; it is a property of the automation.