detection

Two restarts a day at fifteen minutes each is the 2%. The loss decomposed into detection, rescheduling, reload and rewound work, the lever on each, the in-memory checkpoint that makes the interval a minute, the square-root pareto of interval against write cost, and the residual that only fewer failures can remove.

The signature is unusual: many jobs degrade together, throughput collapses, and the drop counters stay at zero. Reading pause counters backward along the paths to find the origin, the three conditions that produce one, and the configuration change that stops it happening again.

It fails open: the data still moves, the job still runs, and everything is about half as fast. The one log line that settles it, the four reasons the library declines to use it, and why the bandwidth signature is close enough to exactly half to be diagnostic on its own.

Roll back first and investigate afterward, because the error budget is being consumed while you read logs. The rollback decision rule that removes the argument, the four causes specific to model serving, and the canary design that would have caught it at one percent of the traffic.

Error correction catches memory faults and reports them. Nothing catches an arithmetic unit that occasionally returns a wrong product, which is why this class is found by comparing results rather than by reading counters. Three detection strategies with their costs, and the signature in a loss curve.

A few lines of comparison, and every difficulty is in the threshold and the clock. Why the collector's receive time rather than the sender's, the boundary case that decides whether an exactly-late rank is reported, and the arithmetic that turns a false-positive rate into a threshold.