TL;DR: Collect device health, performance and telemetry-health signals, then tie alerts to an action and service objective. GPU utilization is limited but can be useful in context; tensor activity, SM occupancy and DRAM activity also have limits. Persistence reduces some noise, but consecutive samples are correlated, so their false-alarm probabilities cannot generally be multiplied.
How to approach it
Start with the decisions: isolate a device, investigate degraded work, or restore missing telemetry. Choose supported fields with known units and reset behavior. Then validate thresholds, collection overhead and notification volume on representative fleet traces.
A strong answer
DCGM (NVIDIA Data Center GPU Manager) is the agent that reads health and performance counters from each GPU and exports them to monitoring. A dashboard pages whenever GPU utilization falls below 90%. It fires during expected checkpoint pauses and misses a job that stays busy while making no useful progress. The repair is not to ban every utilization alert. Define the workload's expected state and combine activity with step or request progress. DCGM and GPU Telemetry explains the device layer; application instrumentation supplies the outcome.
NVIDIA's DCGM feature documentation defines profiling fields. SM occupancy measures resident warps relative to the supported maximum; it is not achieved FLOP utilization. Tensor-pipe activity reports activity over an interval, and DRAM activity is not automatically a direct percentage of peak payload bytes per second. Field availability and simultaneous collection depend on hardware and profiling support. Verify each exported metric's definition and units.
| Signal family | Question it helps answer |
|---|---|
| ECC, remap state, XID events | Is recovery or device investigation required? |
| Link errors and traffic | Is a path degraded or carrying the expected traffic? |
| Temperature, clocks, power reasons | Is a device constrained, and by what condition? |
| SM/tensor/DRAM activity | Which execution resources are active? |
| Exporter freshness and errors | Can the telemetry itself be trusted? |
GPU utilization can stay high for a kernel that uses only a small part of the device. Low activity can be expected during input waits or maintenance. But sustained zero activity while a job is expected to advance, paired with no progress and a tested recovery action, can be actionable. The context decides, not a universal rule that utilization must never alert.
Use counter deltas over valid intervals and account for resets, missing samples and device replacement. Lifetime counts, volatile counts and gauges need different handling. Preserve stable identity and collection timestamps. A missing field is unknown or unsupported, not proof of zero errors. GPU Failure Modes and XID Errors helps interpret events without turning them into certain predictions.
The fleet multiplies weak rules. Under a hypothetical standard-normal one-sided threshold at three sigma, the tail probability is approximately 0.00135:
Devices = 16,384
Expected threshold hits per evaluation = 16,384 × 0.00135 ≈ 22.12
At one evaluation/minute: about 31,850 hits/day before grouping or persistence
Sanity: this expectation does not require independent devices, but the assumed marginal distribution must hold.
Ten consecutive violations are stronger evidence of a persistent condition, but they are not necessarily ten independent observations. Workload, temperature and baseline state often change slowly, so adjacent samples are related. If they were perfectly correlated, requiring ten would leave the crossing probability unchanged. Replay the rule against real traces to measure both the notifications it removes and the delay it adds to detecting an incident.
Separate notification from automated action. A vendor-required recovery state may block new allocation immediately while a successfully recovered isolated event becomes a ticket. Page when human intervention is needed for active impact, failed recovery or material risk. Group related events so one node fault does not create a page per metric.
DCGM diagnostics are workload- and configuration-dependent checks, not fixed-duration guarantees. Use the supported suite for the GPU and failure being investigated. Coordinate intrusive tests with workloads, measure their duration and overhead, and distinguish online telemetry from an acceptance stress test. A passing diagnostic narrows the evidence; it does not guarantee every workload will run correctly forever.
Start with a small actionable set and retain targeted deeper collection for investigations. Reverse a threshold or sampling choice when replay shows excessive false positives, missed incidents or unacceptable collection overhead. Record the reason for each alert alongside its owner and recovery link.
What interviewers probe next
- "Can high occupancy still perform poorly?" Yes; resident warps can wait, and the bottleneck may be elsewhere.
- "What if the exporter stops?" Alert on stale collection using a separate telemetry-health rule.
- "Does persistence eliminate false alarms?" Only to the extent demonstrated by the correlated data and policy.
Common mistakes
- Equating activity fractions with achieved useful throughput.
- Treating unsupported or missing fields as zero.
- Multiplying consecutive tail probabilities as if samples were independent.
Key takeaways
- 16,384 × 0.00135 is about 22.12 threshold hits per evaluation in the stated model.
- A persistent signal is not ten independent observations.
- Combine device metrics with useful work and a defined response.