throttling

A training cluster is a synchronized load, so the peak is far above the average and the average is what everyone budgeted. The immediate mitigation takes minutes and costs a few percent of throughput, and the four durable fixes trade cost against throughput in different places.

Identical replicas serving identical traffic should produce identical numbers, so a difference is a defect and the fleet median is the tool that finds it. Five causes, the three that are about the replica and the two that are about what it was sent.

The exporter offers hundreds of fields and about a dozen of them change decisions. The health fields that predict failure, the performance fields that tell you whether work is happening, the one everybody alerts on that means nothing, and the diagnostic levels with the time each takes.

Five causes produce the same symptom and each has a check that takes under a minute. The order to run them in, what each one eliminates, and the arithmetic showing that one slow node in a thousand costs the whole job its speed rather than a thousandth of it.

Throttling is the hardware protecting itself, so it produces no error and no failure, only a job that is quietly slower. The two telemetry fields that name it, the arithmetic linking clock to throughput, and why the response differs completely depending on whether the cause is one node or the room.

Unlike a training job, a slow serving replica holds nobody else back, so it survives unnoticed while degrading a slice of users. The five causes and the metric that distinguishes each, why routing can create the symptom with no hardware fault, and the fix that limits damage while you investigate.