dcgm

On an HBM part ECC costs almost nothing; on a GDDR part about 6% of capacity and bandwidth. What it buys is the difference between a corrected bit and a silently flipped exponent that turns 1.0 into infinity across a 512-GPU all-reduce. The mechanisms, the signals, and the expected-loss arithmetic.

A training cluster is a synchronized load, so the peak is far above the average and the average is what everyone budgeted. The immediate mitigation takes minutes and costs a few percent of throughput, and the four durable fixes trade cost against throughput in different places.

Compute the step time the formula predicts, measure the one you have, and the gap is the budget to explain. The isolation order with one metric per suspect: dataloader wait, SM and tensor-core activity, HBM throughput, time in collectives, and the per-rank spread that says it is one machine.

Every collective ends when the last rank arrives, so a single throttled GPU taxes 16,383 others. The arithmetic of that tax, the per-rank timing that finds the rank in one step, and the ranked list of causes from a hot GPU to a NIC with symbol errors, each with the command that confirms it.

The number nvidia-smi calls utilization measures whether any kernel was running, not whether the chip was busy. A decode step that saturates HBM can read as thirty percent. The right dashboard has four other numbers on it.

A pre-flight suite is a list of tests with a number on each: DCGM health, a GEMM near the fleet median, NCCL at rated bandwidth, NICs at line rate, links active, storage reachable. What goes in the two-minute gate, what waits for the long diagnostic, and why the gate pays for itself.

The exporter offers hundreds of fields and about a dozen of them change decisions. The health fields that predict failure, the performance fields that tell you whether work is happening, the one everybody alerts on that means nothing, and the diagnostic levels with the time each takes.

GPU utilization at 100% tells you nothing, and a trace per token would cost more than the tokens. The span layout for one request, the GPU counters that read headroom, the cardinality budget that keeps the metrics store alive, and cost attribution that charges tenants for the batch share they used.