preemption

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

Separate quota, borrowing, reclaim latency and fairness, with a demand-conserving example of when pooling improves utilization.

Balance durable checkpoint cost against replay and restart loss, with explicit assumptions and a warning sign when the approximation breaks down.

Size usable inference headroom from the complete readiness delay and demand ramp, then decide which training capacity can safely be borrowed.

Build a durable, bounded scheduling loop with reservations and checkpoint-aware preemption, and avoid universal queue-wait formulas.

Trace GPU credits through reservation, allocation, metering and settlement, with crash-safe ownership and a separate fairness policy.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

A team returns to claim its GPUs, but another team is borrowing them in one indivisible job. Work through the quota tree, whole-job reclaim, recovery costs and release-to-ready budget before promising a start time.

Implement priority admission and completion, preserve a preempted job’s sequence, and distinguish a sufficient greedy victim set from an optimal one.

Simulate admission, decode growth and preemption. Check memory accounting and useful progress with the same finite workload.