drain

Protect running work, validate a supported upgrade bundle, and expand through measured cohorts with a tested recovery path.

Trace a link flap through transport and job recovery. Calculate detection, restart and replay costs, and learn what continuous counters and timed tests can actually establish.

Keep the training state and replace the failed capacity. Trace fault detection, safe checkpoint selection, worker shutdown and restart; size spare capacity from observed failures and repair times.

Treat the upgrade as a tested change to a compatibility set. Drain safely, canary representative workloads, size waves from usable spare capacity and rehearse restoration of the previous working configuration.

Handle an inaccessible GPU with bounded recovery and preserved evidence, without treating a reboot or repeat count as proof of root cause.

Give every health-state transition a reason, owner and completion condition. Calibrate intervention against missed-fault risk and disruption cost, and prevent a bad signal source from removing the whole fleet.