Advanced

Spot, Preemption and Capacity Strategies

Preemptible capacity trades a lower price for interruptions and uncertain availability. Compare useful completed work, billed recovery and capacity wait separately, and restore only a complete checkpoint on a compatible replacement.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Checkpointing and Resumption at Scale Checkpoint I/O Gang Scheduling with Kueue and Volcano Capacity Planning and Utilization TCO: Buy vs Rent Elastic and Fault-Tolerant Training

LESSONS THAT TEACH THIS

Training at ScalePlan retained progress and recovery on one calendar21 min→Fleet and ReliabilityDesign the repair queue and page for urgent unresolved impact30 min→Fleet and ReliabilitySpecify entitlement, borrowing and reclaim under contention20 min→

PRACTICE THIS IN REAL QUESTIONS

Kubernetes, Slurm & GPU SchedulingWe rent GPUs. When should we buy committed capacity instead of paying on demand, and what do we do with the rest of the demand?→Kubernetes, Slurm & GPU SchedulingYour fine-tunes run on spot GPUs preempted about once every four hours. How often should they checkpoint, and when does spot stop paying?→Kubernetes, Slurm & GPU SchedulingOne fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.→GPU Fleet Reliability & ObservabilityAt 16,384 GPUs something fails every three hours. How often should you checkpoint, and what goodput does that leave?→Behavioral & OwnershipTell me about a cost reduction you led. How did you prove reliability did not suffer?→Kubernetes, Slurm & GPU SchedulingEight research teams share 1,024 GPUs. Design the quota and fairness policy, and tell me how they will game it.→