TL;DR: Define the start boundary and percentile before promising sub-second GPU startup. A ready replica avoids initialization on the request path. Cached weights or a process snapshot can reduce work, but restoring device state still moves bytes and rebuilds supported runtime state. Measure warm hits, cached starts and cold misses separately.
How to approach it
Draw the states a request can encounter. Budget provisioning, image access, weight transfer, engine initialization and readiness. Then size ready capacity from traffic and service time, with an explicit queueing model and burst allowance.
A strong answer
A 7B model is called intermittently. The first request after an idle period takes much longer than later requests. Before optimizing, determine whether the platform released only the process, evicted its cached weights, or removed the GPU node entirely. Those are different starting points. Containers, Images and GPU Cold Starts describes the stages; the service promise must name which path it covers.
For an exact 7-billion-parameter BF16 weight budget, weights alone occupy 14 GB in decimal units. Runtime memory, caches and workspaces add more. Suppose a synthetic startup trace has 3 seconds of image access, 1 second of container setup, 2 seconds of weight loading, 30 seconds of engine initialization and 2 seconds of readiness work. Its serialized total is 38 seconds, excluding node provisioning. Real stages can overlap; trace dependencies before summing them.
NVIDIA's CUDA checkpoint utility supports checkpoint and restore behavior that varies with driver version and features. Its documented suspend path copies device memory to host memory and releases GPU resources; resume copies memory back and restores CUDA state. A snapshot can avoid repeating expensive initialization, but it does not mean the weights remained in HBM after the GPU was released.
Assumed device state to restore: 14 GB, excluding other snapshot data
Assumed achieved host-to-device rate: 25 GB/s
Transfer lower bound: 14/25 = 0.56 seconds
At 7 GB/s storage delivery: 14/7 = 2 seconds for that stage alone
Sanity: setup and other state can only add work beyond this simple transfer budget.
A small snapshot already in host memory may restore in under a second on a compatible system. Establish that with a benchmark of the actual application, driver and GPU. The result can change when the snapshot grows, storage must supply it, or a resource cannot be restored by the deployed checkpoint implementation. External connections and other application state also need a safe restart path; restoring GPU memory alone does not make the service ready.
Image layering and lazy access can reduce startup bytes, but do not make image size irrelevant. The working set, storage latency and concurrent reads still matter. Keep weight identity, tokenizer and engine configuration versioned. Cache placement should be weighed against capacity and topology; a perfect cache hit on an unavailable GPU does not serve the request.
Warm capacity is also a queueing decision. As a deliberately simplified loss-system model, assume Poisson arrivals, independent service times averaging 3 seconds, no waiting, and four requests per minute. Offered load is a = (4/60)×3 = 0.2 Erlangs. For c ready single-request replicas, Erlang B gives the probability all are busy:
c=1: B = a/(1+a) = 0.2/1.2 = 16.67%
c=2: B = (a²/2)/(1+a+a²/2) = 0.02/1.22 = 1.639%
Under this model, two ready replicas accept about 98.36% without overflow.
Sanity: mean busy demand of 0.2 does not imply one replica is always available.
This is not a prediction for batched LLM serving, correlated arrivals or a system that queues and scales dynamically. Replay the real arrival and service distributions, including long generations and deployment events. Choose ready replicas and overflow behavior against latency percentiles. An assumed price of $2.50 per GPU-hour makes two one-GPU replicas for eight hours cost $40; shared replicas and billing policies change that budget.
Keep ready capacity when missing the startup objective is expensive. Reverse that choice for tolerant, sparse workloads where a measured cached start is acceptable and idle cost dominates. Report the fraction of requests taking each path and their latency distributions; a fast warm median cannot establish a cold-start p99 promise.
What interviewers probe next
- "Does a snapshot eliminate weight transfer?" Not when device memory was released and must be restored.
- "Can one large request upset the pool model?" Yes; service-time variation, batching and correlated arrivals require a richer model.
- "What happens after eviction?" The request follows a slower measured path or receives an explicit admission response.
Common mistakes
- Calling a warm request a cold-start measurement.
- Promising a restore time without a state-size and bandwidth budget.
- Turning average concurrency into a tail-latency guarantee.
Key takeaways
- 14 GB at 25 GB/s takes at least 0.56 seconds of transfer service.
- Two ready replicas have 1.639% overflow in the stated a=0.2 loss model.
- Price and measure warm, cached and fully cold paths separately.