alerting

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Build observability around durable useful progress and aligned phase timing, showing why full-step timers can hide the late rank.

Define request success, first-output latency and streaming continuity for each service class. Count bad outcomes against an explicit budget, then use paired long and short windows to detect significant ongoing degradation.

Page when timely human action matters. Use budget burn, careful grouping, bounded maintenance rules, and regular testing to keep those pages useful.