concurrency

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

A single throughput number describes a configuration nobody runs. The four variables that have to be fixed, the curve that is the actual deliverable, and the bandwidth bound that turns the result from a description into a diagnosis.

A full cache pool with idle streaming multiprocessors is a specific signature with a specific set of remedies. Six of them, ranked by what they cost, and the one that is free because the flag was set from the model rather than the product.

Concurrency equals arrival rate times residence time, and residence time on an LLM replica grows with concurrency. The chain from 50 requests a second to 1,000 in flight, the replica count with a stated utilization, and the feedback loop that turns 90% utilization into a queue that does not drain.

The bandwidth is effectively unlimited and the first byte takes fifty milliseconds, so throughput is set by how many requests you keep in flight rather than by the network. Little's law applied to a data loader, the request size where the latency stops mattering, and the two design changes that follow.

Two parameters, one lazy refill and one lock. What capacity and rate each control and why conflating them is the usual bug, the clock choice that avoids a whole class of failure, and the retry-after value that turns a rejection into something a client can act on.

The queue is four lines and the shutdown is where it goes wrong. One sentinel per consumer rather than one for all of them, an exception path that cannot silently kill a worker, and a bound that must be small enough to actually apply backpressure.

One atomic increment is correct and does not scale, because every thread fights for the same cache line. The ordering that is sufficient and why the default is stronger than needed, the padding that removes contention entirely, and the arithmetic showing when sharding is worth it.

Two ends, two access patterns, and one design choice that removes almost all contention. Why the owner takes the newest task and thieves take the oldest, what that does to cache locality, and the measurement showing why this cannot be demonstrated in Python at all.