Your fine-tunes run on spot GPUs preempted about once every four hours. How often should they checkpoint, and when does spot stop paying?

Balance durable checkpoint cost against replay and restart loss, with explicit assumptions and a warning sign when the approximation breaks down.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Balance durable checkpoint cost against replay and restart loss, with explicit assumptions and a warning sign when the approximation breaks down.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Spot, Preemption and Capacity StrategiesPreemptible capacity trades a lower price for interruptions and uncertain availability. Compare useful completed work, billed recovery and capacity wait separately, and restore only a complete checkpoint on a compatible replacement.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Multi-Tenancy, Quotas and Fair ShareFollow three teams through GPU borrowing and reclaim. Distinguish quotas, reservations and fair sharing; see why a returning team may leave another above its nominal share, and why failures or indivisible jobs change what can be promised.

Advanced

🧮 Open Weights & Serving Engines🔒 Premium

Pretraining, Fine-Tuning and DistillationUnderstand how a base model becomes a useful application model. Separate training objectives from LoRA and quantization, compare distillation with retrieval, and choose the smallest intervention supported by your evaluation.

Advanced

📐 AI Systems Design🔒 Premium

Model Routing and CascadesChoose a model against measured task quality, latency and cost. Distinguish routing before generation from cascades that pay for a first answer before escalating.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackOne fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.Next in this trackDesign a job queue for 100k GPU jobs with preemption: what state, what ordering, and what happens when a quota owner returns?Next in this trackWhat is Ray on Kubernetes good for, and where do its scheduler and the Kubernetes scheduler fight each other?

FEDITOR'S NOTE

Derive a candidate checkpoint interval from durable-save cost and job-level interruption rate. Keep expected replay, recovery delay and the small-overhead approximation explicit.

DISCUSSION · 0

No comments yet — be the first to share your approach.