TL;DR: Separate ready capacity from instances still starting. Define the cold-start boundary and percentile, reduce initialization work with verified caches, and keep a paid or platform-funded ready reserve where the latency contract requires it. Size that reserve from burst behavior and startup tails. A snapshot is usable only when the complete CPU/GPU/runtime state has a supported restore path.
How to approach it
Ask whether customers submit arbitrary code or a fixed model API, whether GPUs are dedicated or shared, the startup deadline, traffic burst shape and who pays for warm capacity. Draw the instance lifecycle and identify what is ready before a request arrives. Then discuss cache locality, admission, isolation and billing.
A strong answer
An endpoint that has scaled to zero receives twelve requests at once. A dashboard says eighty instances are starting across the fleet. That number tells the caller nothing about whether this model has an available, initialized slot now.
Track requested, placed, initializing, ready, busy and draining instances separately. Before calling a slot ready, verify model execution with the expected artifact version and enough memory for the allowed requests. A running process or healthy HTTP socket cannot establish that. Route only to eligible ready capacity, and use Capacity and Backpressure to bound the queue when those slots run out.
In the toy example, each request occupies one slot and eight slots are ready. Eight start immediately; four wait or are rejected according to the deadline. Separately, 20 instance starts per second with a four-second average startup implies 80 instances in the startup pipeline on average under stable conditions.
Startup arrival rate = 20 instances/s; mean initialization = 4 seconds
Mean instances in initialization = 20 × 4 = 80
Ready reserve in the burst example = 8 available one-request slots
Sanity: 80 starting instances do not make 12 arrivals fit into 8 ready slots.
This is Little's law applied to initialization, not a formula for the ready reserve. Request duration, burst correlation, startup variation and model popularity determine how many ready slots are needed.
Define the cold-start interval from request acceptance to a usable response. Within it, record placement, image availability, runtime initialization, weight transfer and engine warmup. Some stages overlap, so add only work that lies sequentially on the critical path. For example, a 7B BF16 model has roughly 14 GB of weight payload. Moving it at an assumed sustained 7 GB/s takes two seconds before context creation, graph capture, conversion, contention or the first forward pass. Test simultaneous starts too: they share the node’s transfer bandwidth.
Containers, Images and GPU Cold Starts explains how content-addressed image layers and weight files avoid repeated downloads when the right bytes are already near the target GPU. Record cache keys with artifact hashes and engine compatibility. Admit against memory for weights, KV, workspace and runtime overhead; do not classify a model as safe for sharing based on parameter bytes alone. Cache affinity is useful until waiting for the preferred node costs more than fetching on another one.
Snapshotting can remove repeated initialization work, but restoring device memory still moves bytes. NVIDIA's CUDA checkpoint utility provides checkpoint/restore operations for supported CUDA process state; they do not establish that every container, VM, library, collective connection or hardware combination can be restored interchangeably. Validate the exact driver/runtime/device combination and application resources. Do not assume an ordinary microVM snapshot automatically includes a working GPU passthrough state. Keep a tested conventional initialization path when restore is incompatible or fails.
Choose isolation from the trust boundary. Arbitrary customer code needs a host/device isolation design, controlled credentials, network policy and cleanup between tenants. A process boundary or MIG partition is not the complete security model. For a controlled inference API, shared batching can be appropriate after proving request separation, memory bounds and fairness. Hardware profiles constrain feasible partitions; there is no universal small-model size that makes sharing safe.
For a strict interactive endpoint, keep a ready reserve sized from measurements and scale the remaining capacity using queue age and forecast demand. Tolerant batch work can instead wait through startup and benefit from scaling to zero. If the idle reserve costs more than the startup deadline is worth, revisit that product contract. Reporting an average that hides cold starts does not solve the tradeoff.
A billing contract must say whether startup, idle reservation and GPU allocation are charged. Meter acknowledged allocation intervals with stable IDs, reconcile retries and expose startup failures separately. Track startup p50/p95/p99 by cache state, ready-slot misses, deadline-qualified goodput, warm idle GPU-seconds and node-level transfer contention. Test bursts, cache eviction, unavailable preferred nodes and failed restores before promising a percentile.
What interviewers probe next
- How does a new model version get reserve? Warm and verify its own artifact-specific slots before moving traffic, while budgeting the overlap with draining old instances.
- What if snapshot restore fails? Bound the fallback path, record the failure and use conventional initialization. Its longer tail belongs in the startup contract.
Common mistakes
- Calling initializing instances ready or treating Little's law as a percentile guarantee.
- Assuming process snapshots automatically preserve GPU state and external connections across any runtime or device.
Key takeaways
- Eight ready slots admit eight simultaneous one-slot requests; starting instances are a different inventory.
- 20 starts/s × four seconds means 80 starting instances on average, not a zero-wait reserve.
- A 14 GB payload at 7 GB/s has a two-second transfer floor before other initialization work.
- Snapshot compatibility and isolation are explicit engineering contracts, not consequences of using a microVM.