state machine

Give every health-state transition a reason, owner and completion condition. Calibrate intervention against missed-fault risk and disruption cost, and prevent a bad signal source from removing the whole fleet.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

Implement bounded recovery probes, state-generation tickets and a recent failure window. Test concurrency and late completions.

Design job creation, cancellation and listing around retries, durable ownership and an explicit pagination contract.