littles law

Occupancy is a means, and the end is enough bytes in flight to cover memory latency. Little's law gives the number of loads an SM needs outstanding; the register file gives the warps you can afford; the kernels that win on H100 usually run at 25% to 50% occupancy with big tiles and no spills.

Concurrency equals arrival rate times residence time, and residence time on an LLM replica grows with concurrency. The chain from 50 requests a second to 1,000 in flight, the replica count with a stated utilization, and the feedback loop that turns 90% utilization into a queue that does not drain.

The bandwidth is effectively unlimited and the first byte takes fifty milliseconds, so throughput is set by how many requests you keep in flight rather than by the network. Little's law applied to a data loader, the request size where the latency stops mattering, and the two design changes that follow.

Three separate reserves get merged into one number and then argued about. The repair pipeline from Little's law, the restart pool that has to be instantly available, and the correlated-failure buffer sized by the largest thing that can fail at once, each derived and then added.

A replica takes 70 seconds to become useful and the burst arrives in 60, so the autoscaler cannot win the first minute; the warm pool has to. The leading signal, the cooldowns derived from the cold start, the warm-pool size from Little's law and the burst profile, and the cost of every second of cold start left in.