TL;DR: Promise outcomes researchers can plan around: starting eligible jobs within a stated time, preserving recoverable progress and recovering interruptions within a measured budget. Keep API availability as a separate objective where users need it. Define the job class, timing boundaries, ownership and measurement coverage before assigning target numbers. A target without those definitions is difficult to enforce fairly.
How to approach it
Ask which workloads and service promises the platform supports. Define each indicator's numerator, denominator and observation window. Explain how lifecycle events and application progress supply the measurements, then agree what happens when the error budget is at risk.
A strong answer
A service-level objective (SLO) is a measurable promise, such as 99% of eligible jobs starting within 30 minutes, with a defined numerator, denominator and time window. A submission API can be available while a long training job repeatedly loses work. Conversely, a running job can continue through a control-plane outage while users cannot launch another. Both experiences matter. SLOs for AI Systems explains why the objective should describe the service users actually receive, rather than replacing every concern with one uptime number.
Start with a few separately defined objectives:
| User need | Candidate indicator | Boundary to specify |
|---|---|---|
| Start useful work predictably | Fraction of eligible jobs reaching their first useful step within a tier-specific limit | Submission eligibility, quota, resource shape and startup |
| Keep a long run productive | Retained progress or a clearly defined productive-time ratio | Allocation intervals, replay, startup and included step work |
| Recover an interruption | Fraction of covered interruptions recovering within a time limit | Fault occurrence or detection through useful resumed progress |
| Restore trusted state | Successful restore validations and age of the latest recoverable checkpoint | Completion protocol, required state and failure domain |
| Submit and control jobs | Successful eligible API operations | User-visible endpoint, timeout and correctness |
Targets such as 90% productive time or 90% of recoveries within ten minutes are possible proposals, not industry guarantees. Derive them from user needs, traces, failure exposure and the capacity required to meet them. Report distributions by job size and class: an interactive single-GPU session and a 1,024-GPU gang do not have the same placement constraints.
Separate queue wait from startup even if the user-facing objective spans both. Submission to allocation describes one delay; allocation to first useful step includes images, initialization and data preparation. State when a request becomes eligible. Invalid requests and work intentionally held by the user should not silently enter the same denominator as ready-to-run jobs, but publish exclusions so the objective cannot improve merely by reclassifying slow starts.
Productive time needs especially careful accounting. If the numerator includes complete step durations, it already includes waits inside those steps. You cannot then subtract the same loader or collective wait again as a separate loss category. Choose either a retained-step-time accounting measure or a baseline-work model, and explain what it measures. MFU and HFU supplies a different efficiency view; a high stepping-time ratio does not prove efficient kernels.
For a hypothetical job holding a fixed allocation continuously for 30 days, a 90% time-based objective allows 10% × 30 × 24 = 72 hours outside the defined productive state. For a job observed for only 100 allocated hours, the corresponding allowance is ten hours. Do not hand every short job a 72-hour budget or use wall time when the declared denominator is allocated GPU-hours.
Recovery budgets are different. If 100 covered interruptions occur and the objective is that 90% recover within ten minutes, at most ten may exceed that limit under this event-based definition. A percentile dashboard is useful, but the success/failure count makes budget consumption explicit. Record detection, replacement waiting, initialization, restore and first useful progress so the team can locate the delay.
A simple availability model shows how recovery time affects progress. Assume three hours of useful operation between interruptions, no replay and a fixed recovery duration R:
useful fraction = 180 / (180 + R), with times in minutes
at R = 10: useful fraction = 94.74%
at R = 9: useful fraction = 95.24%
improvement = about 0.50 percentage points
This is a scoped model, not a fixed claim that every saved recovery minute adds the same percentage of goodput. Add replay, checkpoint overhead and other losses only under consistent boundaries. Checkpointing and Resumption at Scale explains the state needed to preserve progress.
Submission → first useful step
- Measure queueing and startup separately
Interruption → recovered progress
- Include detection, placement and restore
Committed checkpoint → restore test
- Verify required state and data position
For checkpoints, distinguish successful publication, readable bytes and valid restoration of the intended training state. A periodic restore drill samples those properties; it does not prove that every checkpoint will survive every storage failure. Define the durability design and recovery-point objective, including replication lag, corruption detection and which failures the system is intended to tolerate. A guarantee of no more than one checkpoint interval lost requires mechanisms and evidence beyond a recent successful test.
Agree an error-budget policy with users and platform owners. When losses exceed the budget, reduce avoidable risky changes and prioritize the largest demonstrated contributors. Preserve exceptions for urgent repair and security work; deferring all maintenance can worsen the incident rate. Google's example error-budget policy illustrates how an agreed policy connects service reliability to release decisions.
Keep allocation, utilization and spare capacity visible as planning inputs. Maximizing utilization alone can conflict with fast recovery or short queue wait, but unused capacity is not automatically justified reserve either. Explain what service objective each reserve supports and measure whether it does so.
For short independent jobs, success rate and completion deadline may matter more than per-job productive-time ratios. Use the same lifecycle evidence with an objective suited to that workload. Revise a target when its cost and user value do not match, through an explicit agreement rather than a hidden metric-definition change.
What interviewers probe next
- "Whose code caused the delay?" Preserve the user-experience measurement, then attribute platform and application causes under the declared policy.
- "What does a missing event do?" Mark measurement coverage and reconcile lifecycle state; do not automatically call missing data success.
- "Does a healthy API prove training availability?" It proves only the measured API behavior, so track job outcomes separately.
Common mistakes
- Rejecting API availability even when submission and control are part of the service.
- Counting the same in-step wait in both productive time and the loss breakdown.
- Treating a sampled restore as proof of universal durability.
- Freezing urgent remediation merely because an error budget is exhausted.
Key takeaways
- Define each objective's population and timing boundaries before its target.
- A 90% time objective over 100 allocated hours permits ten hours of defined loss.
- Error budgets should guide agreed decisions about user-visible outcomes.