dcgm

Trace a changed BF16 bit, distinguish corrected and uncorrectable errors, and calculate recovery cost without assuming every device has the same ECC overhead.

Investigate rack power events at the correct electrical boundary and time scale. Test load reductions without assuming GPU caps control every watt or transient.

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.

Separate a longer step from lower throughput, trace the dependency that changed, and distinguish input, workload, device and path causes. Then calculate whether a repair recovers more time than it costs.

Thirty percent GPU utilization means kernels ran during about thirty percent of the sampled time. Find out what limits those kernels, what happens between them, and whether requests are meeting their latency targets.

Use a short admission check for recent changes and a deeper acceptance suite for new or repaired nodes. Set thresholds from matched healthy baselines, and make quarantine and return-to-service decisions explicit.

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Trace a stalled stream without unbounded telemetry, understand what sampling loses, and reconcile GPU allocation with a stated cost policy.