One fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.

A shared pool recovers the GPUs inference holds for its peaks, but a preempted training gang takes minutes to give them back and an SLO breaks in seconds. The utilization of each design, the reclaim-time arithmetic against the traffic ramp, and the floor-plus-borrow split most fleets land on.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Multi-Tenancy, Quotas and Fair ShareA shared GPU pool is cheaper than ten private ones because ten teams' demand is smoother than one team's, and it only works if the sharing is enforced. Quotas say what each team is guaranteed, borrowing lets idle guarantees be used by others, fair share decides who waits when everyone wants more, and preemption reclaims borrowed capacity. This page works the arithmetic that makes pooling worth it, the layers of isolation a tenant needs, and the incentive problems (hoarding, gaming, the research-versus-product tension) that any policy has to survive.

Core

🩺 Fleet Reliability & ObservabilitySign in

SLOs for AI SystemsA service level objective is a promise with a number attached, and AI systems need their own because the classic ones do not fit: a training run has no requests, only progress, so its objective is goodput; an LLM endpoint streams, so its latency is two numbers (time to first token and time per token) rather than one; and both spend a budget that is set by hardware failure rates rather than by software bugs. This page defines the objectives that fleet and serving teams actually use, derives the thresholds from user needs and from the hardware, and works the error-budget arithmetic that decides when to stop shipping and start fixing.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Spot, Preemption and Capacity StrategiesSpot and preemptible GPUs cost a fraction of on-demand and can be taken back with a couple of minutes' notice, so using them well is an expected-value calculation: the discount against the work lost per preemption, which is set by checkpoint cadence and restart time. The same arithmetic governs internal preemption in a shared cluster. This page works the break-even, the checkpoint interval that makes spot pay, and the fleet mix (reserved baseline, on-demand headroom, spot for tolerant work) that a capacity strategy is built from.

Core

🗂️ Scheduling & OrchestrationSign in

MIG, MPS and Time-SlicingA whole H100 is far more than a notebook, a small inference service or a CI job needs, and giving each of them a card leaves most of the fleet idle. Three mechanisms share a GPU, and they differ in what they isolate: MIG partitions the hardware into up to seven slices with their own memory and compute, MPS lets several processes share one GPU's SMs concurrently with no memory isolation, and time-slicing context-switches between processes with no isolation at all. The choice is the isolation the workload needs against the utilization the platform wants.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign a job queue for 100k GPU jobs with preemption: what state, what ordering, and what happens when a quota owner returns?Next in this trackWhat is Ray on Kubernetes good for, and where do its scheduler and the Kubernetes scheduler fight each other?Next in this trackOur GPU pods take four minutes to start on a fresh node and sometimes fail with 'no CUDA-capable device'. Walk me through both.

FEDITOR'S NOTE

Scored on computing what the separate-pool design wastes and what the shared design risks, on knowing that reclaim time is minutes while an SLO breach is seconds, and on the floor-plus-borrow design with the ramp-rate condition that makes it safe.

DISCUSSION · 0

No comments yet — be the first to share your approach.