autoscaling

A replica takes a minute to become useful and the traffic changes in seconds. Scale on the wrong signal and you buy GPUs you never use, or you drop requests while they boot. The design is three numbers and two timers.

Ten thousand requests a second is a fleet of hundreds of nodes, a router that has to know what every replica is caching, and an SLO pair that decides the batch on every one of them. Here is the sizing chain, node by node.

Twenty models of different sizes is a packing problem with a cold-start penalty, and the two pull in opposite directions. What stays resident, what loads on demand, the arithmetic that decides which is which, and the interface that stops every team from asking for a dedicated replica.

Peak sets the fleet, average sets the bill, and the ratio between them is idle money. The chain from a 3x diurnal peak to a replica count, the utilization it implies, what autoscaling can and cannot recover given model load times, and what to do with the trough.

Ray gives you actors, tasks and placement groups inside a cluster of pods that Kubernetes thinks are just pods. Two schedulers, two views of the same GPUs: the idle-node arithmetic, the placement group that cannot be gang-scheduled by the layer below, and the rules that stop the two from wasting each other's capacity.

A plain Deployment reports ready before it can serve and scales on a signal that never moves. What an inference operator adds: weight delivery, readiness tied to warm-up, scaling on queue depth with cold starts accounted for, a prefix-aware router, and rollouts that keep streams alive. KServe and llm-d as of 2026.

Seven boxes between an API call and a GPU, each with one job and one way to fail. The walkthrough a screen expects in the first ten minutes, with the sizing chain from 2,000 concurrent users to a replica count so the drawing has numbers on it.

A replica takes 70 seconds to become useful and the burst arrives in 60, so the autoscaler cannot win the first minute; the warm pool has to. The leading signal, the cooldowns derived from the cold start, the warm-pool size from Little's law and the burst profile, and the cost of every second of cold start left in.