One fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.

Size usable inference headroom from the complete readiness delay and demand ramp, then decide which training capacity can safely be borrowed.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Size usable inference headroom from the complete readiness delay and demand ramp, then decide which training capacity can safely be borrowed.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Multi-Tenancy, Quotas and Fair ShareFollow three teams through GPU borrowing and reclaim. Distinguish quotas, reservations and fair sharing; see why a returning team may leave another above its nominal share, and why failures or indivisible jobs change what can be promised.

Advanced

🚀 Inference & Serving🔒 Premium

Reasoning Budgets and Answer LatencyReasoning tokens consume generation time and retained state before an answer becomes usable. Separate the first generated token from the first answer token, reserve output capacity, and evaluate reasoning limits against task success.

Advanced

📐 AI Systems Design🔒 Premium

Offline Batch Inference vs Online ServingOnline serving promises each request a latency; a batch job promises a deadline for a whole set and a checked result for every item. That changes what to optimize, when an engine's startup cost dominates, and what delivery means.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Spot, Preemption and Capacity StrategiesPreemptible capacity trades a lower price for interruptions and uncertain availability. Compare useful completed work, billed recovery and capacity wait separately, and restore only a complete checkpoint on a compatible replacement.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign a job queue for 100k GPU jobs with preemption: what state, what ordering, and what happens when a quota owner returns?Next in this trackWhat is Ray on Kubernetes good for, and where do its scheduler and the Kubernetes scheduler fight each other?Next in this trackOur GPU pods take four minutes to start on a fresh node and sometimes fail with 'no CUDA-capable device'. Walk me through both.

FEDITOR'S NOTE

Measure from the reclaim decision to inference readiness. Evicting a training pod is only one part of that interval.

DISCUSSION · 0

No comments yet — be the first to share your approach.