automation

A check that is too strict removes healthy capacity and one that is too loose lets a broken node take a job down. The three tiers by cost, the fleet-median comparison that catches degradation no threshold does, and the false-positive budget that has to be set before any of it runs.

Firmware skew produces intermittent faults that surface layers away from their cause, so an inventory is a debugging tool rather than a compliance exercise. What to collect, how drift appears without anyone deploying anything, and the diff that turns a week-long investigation into a query.

You cannot remove one rank from a synchronous job; you replace the node and restart from checkpoint, and the whole craft is making that take three minutes instead of thirty. The detection signals, the drain-restart-quarantine sequence, the spare-pool arithmetic, and where elastic training changes the answer.

The driver logs a numbered code when something goes wrong on a GPU, and the number tells you whether to retry the job, drain the node or file a hardware return. The codes worth memorizing in three groups, the action each implies, and the automation that turns a log line into a drained node.

The device has stopped answering on PCIe, so the driver cannot reset it and no software fix exists: the host has to re-enumerate the bus. What causes it, why the node is out until it reboots, the three-strike policy that separates a transient from a dying card, and the automation that handles it without a human.

A fleet of ten thousand GPUs generates enough hardware events to page someone hourly, and a rota that receives them stops reading them within a month. The three tests an alert must pass to page, the arithmetic of a sustainable rota, and the automation that has to exist first.

The controller's real design problem is not detecting faults but deciding when a signal is strong enough to remove a node from a job. The six states and their transitions, the cost asymmetry that sets every threshold, and the arithmetic showing what a one-percent false-positive rate costs at fleet scale.

A cold plate holds a few tens of grams of water against seven hundred watts, so temperature rises about two degrees a second and the limit arrives in under twenty. What that timescale rules out, the protection layers that act inside it, and the choice that decides whether you lose a job or the hardware.

Measure the hours before you rank anything, because the task that feels worst is usually not the one that costs most. The ordering rule that keeps automation from causing the outage it was meant to prevent, and why detection always ships before remediation.