TL;DR: A cold start is a chain: schedule, pull the image, start the container, load the weights, initialize the engine, warm up. Naive, that is minutes; the platform's job is to remove links. Lazy image loading cuts the pull to seconds, a per-node weight cache cuts the load to a few seconds, a memory snapshot of the initialized process (host and GPU memory) turns initialization into a restore measured in hundreds of milliseconds, and a warm pool sized from the arrival rate hides the rest. Sub-second means the request lands on a restored snapshot, never on a fresh container.
How to approach it
Ask what the function does (a 7B model in bf16 is 14 GB of weights; that number drives everything), what the arrival pattern looks like (bursty functions need warm pools, steady ones do not), and whether users bring arbitrary images. Then write the chain on the board with a naive number per link and say which links you delete rather than shorten. Close with the pool arithmetic, because "keep some warm" without a number is not a design.
A strong answer
A typical situation: a platform lets users deploy a Python function decorated with a GPU requirement. A user's function loads a 7B chat model, is called a few times a minute during the day and not at all overnight, and the user complains that the first call after any gap takes three minutes.
The chain, with its naive cost and the fix per link:
7B model, bf16: 7e9 × 2 B = 14 GB of weights; image ~12 GB (CUDA, PyTorch, the engine)
link naive fix tuned
1 schedule 0 to 120 s (add a node) keep a pool of GPU nodes 0 s
2 image pull 12 GB at 150 MB/s = 80 s lazy-load snapshotter, content- 2 to 5 s to first exec
addressed layer cache on node
3 container start ~1 s (toolkit injects unchanged ~1 s
driver libs)
4 weight load 14 GB from object storage weights on node NVMe at ~7 GB/s 14 ÷ 7 = 2 s
at 1 GB/s = 14 s or pinned host memory over PCIe ~0.5 s
5 engine init graph capture, JIT, persistent compile cache, then 0.3 to 0.8 s
allocator: 30 to 90 s snapshot restore of the whole
initialized process
6 warm-up first request ~5 s run one canned request before 0 s on request path
marking ready
naive total: 80 + 1 + 14 + 60 + 5 ≈ 160 s (plus a node add if unlucky)
tuned, no snapshot: 3 + 1 + 2 + 30 ≈ 36 s
tuned, snapshot restore: the restore replaces links 2 to 6 → ~0.5 to 1 s
sanity: every link except restore is bounded below by moving 14 GB somewhere; only a
restore from memory that already holds the model gets under a second
Containers, Images and GPU Cold Starts covers each link; the platform design is which links it owns.
Images. The platform builds images itself from the user's declared dependencies, so it controls layering: a shared CUDA and framework base that is already on every node, and a thin user layer on top. A lazy-loading snapshotter (eStargz or SOCI style) starts the container after fetching only the files the interpreter touches, so link 2 becomes seconds regardless of image size. User-supplied images lose this unless they are re-based onto the shared layers at deploy time.
Weights. Weights never go in the image. The platform keeps a content-addressed weight cache on each node's NVMe, prefetches the weights for functions likely to land there, and streams from a regional cache when they are absent. A loader that maps safetensors shards straight to GPU memory lands 14 GB from NVMe in about two seconds.
Snapshots. This is the link that gets under a second. After the function's first cold start, the platform checkpoints the initialized process: host memory through a CRIU-style process checkpoint, GPU memory through the driver's checkpoint and restore support. The snapshot is stored and distributed to nodes with the right GPU type. A later cold start restores the snapshot: the process resumes with the model already in HBM and the CUDA context rebuilt, in hundreds of milliseconds. The constraints are real: the restore must land on the same GPU model and a compatible driver, the snapshot is tens of gigabytes, and anything the process holds that cannot be snapshotted (open network sockets, a running background thread that expects the old clock) has to be handled by the platform's runtime shim.
Warm pools. Even a restore takes half a second and a GPU; the pool decides how often a request pays it. The size comes from the arrival rate and the start time:
inputs: arrival rate λ = 4 requests per minute at peak, service time s = 3 s per request
restore time c = 0.5 s; target: 95% of requests hit a warm container
concurrency needed for the steady load = λ × s = 4 × 3 s ÷ 60 s = 0.2 containers busy on average
Poisson arrivals at this load: with 1 warm container idle, the chance it is busy when the next
request lands ≈ 0.2, so 80% warm; with 2 warm, the chance both are busy ≈ 0.2² ÷ 2 ≈ 0.02, so 98%
keep 2 warm at peak; scale to 0 after an idle window of, say, 10 min, and rely on restore after
cost of the pool: 2 GPUs × $2.50 × 8 peak hours = $40 per day for this function
sanity: a function called 4 times a minute needs about one fifth of a GPU; two warm containers
is 10x its average need, which is the price of a 98% warm hit rate on bursty traffic
The pool is per function and per GPU type, and it is what the platform bills its margin on. Scale-to-zero plus snapshot restore is the design that makes a cheap tier possible: after the idle window the GPU is released, and the next request pays 0.5 s rather than 36 s.
The condition that reverses the design: a 70B model in bf16 is 141 GB of weights across multiple GPUs, and its snapshot is that size plus host state; restoring it is tens of seconds even from NVMe, so the sub-second promise holds only for models that fit one GPU and whose snapshot is small enough to keep resident. For large models the platform keeps a minimum of one replica running and the pool is the answer, not the snapshot.
The reversal condition: a model small enough that the naive path already meets the SLO. Under a few billion parameters the weights load in seconds and the sub-second target is achievable without snapshots or lazy loading at all. Containers, Images and GPU Cold Starts has the per-stage costs that decide it. Inference Autoscaling and Cold Starts is the serving-side view of the same chain.
What interviewers probe next
- "A user's function opens a database connection at import time. What does the snapshot do to it?" The socket does not survive restore; the runtime shim has to reconnect on resume or the platform documents that connections belong in the request handler.
- "Why not snapshot at build time instead of after the first cold start?" You can, on a build node with the same GPU type; it removes the first-user penalty at the cost of a GPU-minute per deploy.
- "How do you place a restore?" On a node that already holds the snapshot and the weights and has a free GPU of the right type; the scheduler scores cache locality above spreading.
- "What makes the p99 worse than the p50 here?" Requests that miss the snapshot cache and pay the fresh start, and requests that land during a node's image or snapshot eviction.
Common mistakes
- Optimizing the image size first; the weight load and initialization dominate, and lazy loading makes image size nearly irrelevant.
- Promising sub-second for every model size; a 141 GB snapshot cannot restore in a second.
- Sizing the warm pool by intuition rather than from λ × service time and the target hit rate.
- Forgetting that a GPU snapshot is tied to a GPU model and driver version, so the fleet's compatibility matrix is part of the design.
Key takeaways
- The chain: schedule, pull (12 GB), start, load (14 GB), init, warm-up; naive about 160 s, tuned about 36 s, snapshot restore under a second.
- Delete links: lazy-load images, cache weights on NVMe, snapshot the initialized process including GPU memory, warm-pool the rest.
- Warm pool size from λ × service time and the target hit rate; two warm containers give about 98% at 4 requests a minute.
- Sub-second holds for models that fit one GPU; large models keep a minimum replica instead.