TL;DR: Fleet reliability includes GPU, host, network, storage and software events. The Llama 3 report provides a useful job-interruption observation: 419 unexpected interruptions over a 54-day period on 16,384 GPUs. That normalizes to about 1.97×10⁻⁵ interruptions per GPU-hour, but it is not an individual GPU failure hazard or a replacement rate.
How to approach it
Define the event being counted before calculating a rate. Separate job interruption, device quarantine and physical repair. Use published exposure as a planning example, then replace its assumptions with fleet measurements and size recovery capacity from the relevant repair workload.
A strong answer
A training team expands its fleet and finds manual recovery consuming more time. More devices and dependencies can create more opportunities for interruption, but the rate depends on the workload, topology and failure domains (sets of components that fail together, such as every node behind one switch or one power feed). Training Uptime and Interruption Statistics offers a public anchor, not a universal reliability specification.
The Llama 3 paper reports 466 total interruptions in that observation period: 47 planned and 419 unexpected. About 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues. It also reports more than 90% effective training time and only three cases requiring significant manual intervention. Those facts show that frequent interruptions and high useful uptime can coexist with effective recovery.
Observed duration = 54 × 24 = 1,296 hours
Unexpected interruption rate = 419/1,296 ≈ 0.3233 per hour
Mean spacing over that record = 1,296/419 ≈ 3.093 hours
Exposure = 16,384 × 1,296 = 21,233,664 GPU-hours
Normalized interruption rate = 419/21,233,664 ≈ 1.9733e-5 per GPU-hour
Sanity: the numerator counts job interruptions, not failed individual GPUs.
One interruption might come from a switch, a host or a software dependency; another might involve several GPUs. Dividing the count by GPU-hours makes records of different sizes easier to compare. It does not tell you how long an individual GPU lasts. The hardware-attribution percentage also says nothing on its own about how many events had a warning or required a replacement.
For an explicit linear planning model using rounded rate 2×10⁻⁵ per GPU-hour, 512 GPUs give 0.01024 interruptions/hour, or one every 97.66 hours. At 4,096 GPUs the model gives 0.08192/hour, or one every 12.21 hours. Label these as extrapolations. Correlated rack events, different job sizes, aging and changes in operations can make linear scaling inaccurate.
Build the response around what is affected. GPU Failure Modes and XID Errors helps classify observed driver events; host, fabric, storage and application signals fill the rest of the picture. Correctable counters can justify investigation, but not every fatal event has a warning and not every trend predicts failure. Preserve uncertainty in incident attribution rather than assigning every timeout to the nearest device error.
A recovery controller should detect loss of useful progress, preserve evidence, isolate unsafe capacity, restart supported jobs from durable state and verify resumed progress. Escalate when recovery fails, repeated events threaten capacity or data correctness is uncertain. Automation should not silently retry a deterministic application failure forever.
Spare capacity follows a different numerator. Suppose, hypothetically, two eight-GPU nodes enter repair each day and average repair time is three days. Mean repair inventory is 2×3 = 6 nodes, or 48 GPUs, under a stable flow. That average is not enough for a tail guarantee; add correlated-loss and maintenance scenarios, compatible topology and a chosen service objective. Do not derive a universal 2–3% spare pool from all job interruptions.
Cost also follows the affected job, not necessarily the whole fleet. A 1,024-GPU job losing twenty minutes incurs 341.33 GPU-hours of replay/recovery exposure. At an assumed $2.50 per GPU-hour, that is about $853.33. If only that job stops, multiplying by every GPU in the fleet exaggerates the cost.
Use the published record to motivate measurement and automated recovery. Reverse any capacity or cadence estimate when local event definitions, repair arrivals or recovery times differ. Keep separate dashboards for interruption frequency, useful time lost, devices out of service and repeat offenders.
What interviewers probe next
- "Does an interruption imply damaged hardware?" No; it describes disrupted job progress.
- "Can one fault stop several jobs?" Yes, so deduplicate the infrastructure incident while retaining each job's impact.
- "What replaces the public rate?" Local exposure and event histories with consistent definitions.
Common mistakes
- Converting normalized interruption exposure into device life expectancy.
- Assuming a hardware-attributed event was predictable or required replacement.
- Sizing repair spares from every restart event.
Key takeaways
- 419/1,296 gives about 0.323 unexpected interruptions per hour in the cited record.
- Two nodes/day with three-day repair time means six nodes in repair on average.
- Count job impact, device health and repair demand separately.