AI Infra Interviews logo
Kubernetes, Slurm & GPU Scheduling / 17
mediumNewCoreWeaveLambdaAWS

We rent GPUs. When should we buy committed capacity instead of paying on demand, and what do we do with the rest of the demand?

Committed capacity is cheaper per hour and cannot shrink, so the decision is a utilization break-even, not a discount. The arithmetic for where the reserved baseline should sit, why the p50 of steady demand is the target, and how on-demand, spot and capacity blocks absorb everything above it.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Committed capacity is cheaper per hour and cannot shrink, so the decision is a utilization break-even, not a discount. The arithmetic for where the reserved baseline should sit, why the p50 of steady demand is the target, and how on-demand, spot and capacity blocks absorb everything above it.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced
🗂️ Scheduling & Orchestration🔒 Premium
Spot, Preemption and Capacity StrategiesSpot and preemptible GPUs cost a fraction of on-demand and can be taken back with a couple of minutes' notice, so using them well is an expected-value calculation: the discount against the work lost per preemption, which is set by checkpoint cadence and restart time. The same arithmetic governs internal preemption in a shared cluster. This page works the break-even, the checkpoint interval that makes spot pay, and the fleet mix (reserved baseline, on-demand headroom, spot for tolerant work) that a capacity strategy is built from.
Advanced
🧮 Napkin Math & Capacity🔒 Premium
Capacity Planning and UtilizationCapacity planning for GPUs is deciding how many to have next quarter given that they cost money whether busy or not, that demand arrives in bursts, and that a queue near saturation produces waits that grow without bound. This page works the planning arithmetic for a serving fleet (peak demand, headroom, the p99 penalty of running hot) and a training platform (job mix, queue wait, the value of a shared pool), and gives the queueing intuition that makes 70% look full. The number that decides everything is utilization, and it has a ceiling set by latency, not by hardware.
Core
🗂️ Scheduling & OrchestrationSign in
MIG, MPS and Time-SlicingA whole H100 is far more than a notebook, a small inference service or a CI job needs, and giving each of them a card leaves most of the fleet idle. Three mechanisms share a GPU, and they differ in what they isolate: MIG partitions the hardware into up to seven slices with their own memory and compute, MPS lets several processes share one GPU's SMs concurrently with no memory isolation, and time-slicing context-switches between processes with no isolation at all. The choice is the isolation the workload needs against the utilization the platform wants.
Foundational
🧮 Open Weights & Serving Engines
Capacity Planning for Open-Weights FleetsPlanning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.
UP NEXT ON YOUR JOURNEY
FEDITOR'S NOTE

Scored on the break-even derivation (reserved is cheaper only above a utilization you can compute from the two prices), on sizing the reserved baseline to steady demand rather than peak, and on naming what handles the rest.

DISCUSSION · 0

No comments yet — be the first to share your approach.