throttling

Investigate rack power events at the correct electrical boundary and time scale. Test load reductions without assuming GPU caps control every watt or transient.

An apparent outlier can reflect workload mix, queueing, runtime state or a degraded execution path. Normalize the comparison, run a matched replay and use critical-path arithmetic to test possible causes.

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Find the critical-path delay using local phase timing and controlled comparisons, rather than diagnosing a node from utilization or clock ratios.

Correlate clock-event reasons, temperature, configured limits and workload timing. Use the scope of the slowdown to choose the next investigation, and distinguish intentional power limits from unexpected thermal constraints.

Compare equivalent requests, inspect the replica’s software and device state, and reduce new traffic only after checking where that load will go.