checkpointing

Compare host-staged and direct GPU-to-NIC transfers, count traffic at each boundary, and check transport selection before blaming a slow collective on missing GPUDirect.

Balance checkpoint pauses against lost work using job-level interruption data, while accounting for asynchronous durability lag.

Describe logical tensors independently of writer ranks, publish only a complete checkpoint, and reconstruct the target layout from stored chunks. Asynchronous saving reduces blocking but still needs a consistent snapshot and a memory budget.

Inventory a resumable checkpoint instead of rounding its bytes per parameter. Trace snapshot stall, durable drain and publication, then calculate the storage floor.

Convert useful tokens into input traffic, then calculate separate checkpoint and restore budgets. Learn why an asynchronous writer can keep up with the cadence and still miss the recovery deadline.

Estimate activation memory from saved tensor shapes and lifetimes. Use a published reference formula carefully, then check checkpointing and parallel ownership against the real peak.

Balance durable checkpoint cost against replay and restart loss, with explicit assumptions and a warning sign when the approximation breaks down.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.