Alert Design and On-Call Load
Useful alerts identify a condition that needs a timely response. Connect pages to user impact or imminent risk, and reduce duplicate noise without hiding failures.
TL;DR: Choose alert conditions with a clear owner, urgency and action. SLO burn rates can reveal sustained impact; diagnostic signals add context or warn of imminent failure. Group related notifications and scope maintenance suppression carefully. Review actionability and missed incidents together when improving the rotation.
Symptom, not cause
A fleet has many causes and few symptoms. Pages should sit on the symptoms, with the causes available as context once someone is looking.
| Alert on this symptom | Rather than these causes |
|---|---|
| Training step time above its threshold for N minutes | A degraded NVLink, a thermal cap, a slow rail, a straggler rank |
| Job restart rate per queue above baseline | Any individual node failure |
| Serving p99 latency breaching its objective | Cache pressure, a slow replica, a routing skew |
| Jobs unable to start for N minutes | Quota, capacity, image pull, a health-check loop |
| Fleet capacity below a floor | Individual nodes going out of service |
why cause-based alerting fails on a GPU fleet
at the published failure rate of about 2 x 10^-5 per GPU-hour, a 2,048-GPU fleet sees
2,048 x 24 x 2e-5 = about one interruption per day under this illustrative normalized rate, not a measured hardware failure rate
most are absorbed: a job restarts, the scheduler places elsewhere, nobody needed to act
paging on each one produces roughly 30 pages a month for events that required no decision
sanity: the same fleet paging on symptoms produces a page only when a job actually could not
proceed, which is a much smaller number and every one of them has an action
Burn rate rather than instantaneous breach
an objective of 99.5 percent over 30 days
error budget = 0.5 percent of 30 days = 3.6 hours
instantaneous alerting: page whenever the current window is below 99.5 percent
a two-minute blip breaches it and pages, and nothing needed doing
burn-rate alerting: page on how fast the budget is being consumed
a 14.4x burn rate exhausts 30 days of budget in about 50 hours: page immediately
a 6x burn rate exhausts it in about 5 days: page during working hours
a 1x burn rate is the objective being met exactly: no page
the two-window form
require the burn rate to hold over both a long and a short window, so a spike that
recovers does not page and a sustained trend does
sanity: this converts "is it broken right now" into "will this cost us the objective", which
is the question a person being woken can actually act on
SLOs for AI Systems covers defining the objective the burn rate is measured against.
Begin with the decision the signal enables, including urgent hardware or facility protection.
No immediate human action
- Record or route for follow-up
Use a dashboard, ticket or working-hours alert appropriate to the impact.
Urgent action
- Group related notifications
Keep affected scope and unexpected symptoms visible.
- Check maintenance scope
Suppress only the expected condition within the approved window; preserve unexpected impact.
- Notify with evidence and a runbook
Make the required first decision clear.
The GPU-specific false alarms
Four signals look like failures on a fleet and usually are not, and an alerting system that has not accounted for them will page constantly.
- Corrected memory errors. Single-bit errors are corrected by design and occur continuously across a large fleet. The alert belongs on the rate rising against its own baseline, or on remapping capacity approaching exhaustion, not on any individual event.
- Driver-level errors caused by user code. Some error classes are raised by a job's own kernel doing something invalid, so they indicate a broken job rather than broken hardware. Routing them to the job owner rather than to the on-call engineer is the fix.
- Utilization dropping to zero. Checkpoints, evaluation phases and data loader stalls all produce it, and a threshold on instantaneous utilization pages during normal operation. Pair it with progress, meaning the step counter, before it means anything.
- A node briefly unreachable. Reboots and agent restarts happen. Require the condition to persist before it becomes a page.
Measuring the rotation
Track how often a page led to a useful action, how many interruptions each shift receives, and whether the same unresolved condition pages repeatedly. Also review incidents that were missed or detected too late. A quiet pager is not enough if users discover failures first.
Downgrade or remove alerts that do not need an immediate decision, while preserving urgent hardware and facility warnings where early action prevents harm. Group related notifications so one incident does not become dozens of pages. Judge the result from both operator load and detection quality; no fixed actionable-page percentage settles that tradeoff.
What interviewers are listening for
Explain why the condition needs a person now and what decision the page enables. Distinguish a single corrected error from a concerning trend, and a planned pause from lost progress. Check that grouping and suppression preserve unexpected impact. Actionable-page rates help assess noise, but they are not the only measure of alert quality.
Key takeaways
- Alert on symptoms a user or job notices; causes belong in the context, not on the pager.
- Page on error-budget burn rate over two windows, so a blip is silent and a trajectory is loud.
- A 2,048-GPU fleet sees about one hardware event a day that needs no human action, which is why cause-based paging fails.
- Account for the four GPU false alarms: corrected memory errors, user-caused driver errors, utilization dips at checkpoints, and brief unreachability.
- Track the fraction of pages that led to an action; review it alongside missed incidents, detection delay and pages per shift.