cold start

A replica takes a minute to become useful and the traffic changes in seconds. Scale on the wrong signal and you buy GPUs you never use, or you drop requests while they boot. The design is three numbers and two timers.

141 gigabytes have to move from somewhere to eight GPUs, and every hop has a bandwidth. Add the CUDA init, the engine warm-up and the graph capture, and the minute is gone unless you design each step.

Twenty models of different sizes is a packing problem with a cold-start penalty, and the two pull in opposite directions. What stays resident, what loads on demand, the arithmetic that decides which is which, and the interface that stops every team from asking for a dedicated replica.

A naive cold start for a 7B function is minutes: pull, start, load, initialize. Under a second means deleting links, not speeding them up: lazy image loading, weights from a local cache, a snapshot of the initialized process with its GPU memory, and a warm pool sized from arrival rate. The chain with a number per link.

Four minutes is a 15 GB image pulled through one registry connection; the CUDA error is a container runtime newer than the host driver. The pull arithmetic layer by layer, what a cache hit actually saves, the compatibility rule with its two exceptions, and the image layout that makes both problems go away.

A serverless GPU is a promise to run a customer's model within seconds of a request while charging only for the seconds it runs. The cold-start chain from image pull to first token, the snapshot that collapses it, the warm pool sized by Little's law, and the isolation and packing decisions that set the margin.

A replica takes 70 seconds to become useful and the burst arrives in 60, so the autoscaler cannot win the first minute; the warm pool has to. The leading signal, the cooldowns derived from the cold start, the warm-pool size from Little's law and the burst profile, and the cost of every second of cold start left in.