concurrency

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Define a reproducible workload and service contract, account for every offered request, and choose a measured operating region with evidence and margin.

Confirm that requests are waiting for usable state capacity. Separate allocated memory, live blocks and evictable prefixes, then compare admission, lower-precision state, prefix sharing and hardware changes with explicit assumptions.

Use Little’s law to distinguish waiting and active requests, then test saturation with a stated service model. Avoid turning a mean occupancy identity into a universal latency rule.

Calculate how many reads a training loader needs in flight, then see when larger requests waste bandwidth. Worked examples separate useful samples, fetched bytes, prefetch and the limits of Little's Law.

Two parameters, one lazy refill and one lock. What capacity and rate each control and why conflating them is the usual bug, the clock choice that avoids a whole class of failure, and the retry-after value that turns a rejection into something a client can act on.

Implement a bounded producer-consumer pipeline with explicit shutdown and error reporting. Keep work outside collection locks and account for memory beyond the input queue.

Implement shared and sharded C++ counters, explain relaxed ordering and test final counts without promising unmeasured scaling.

Implement owner-pop and thief-steal semantics with correct accounting for running tasks that generate children.

Keep request timing independent, make snapshots consistent, and expose overwritten records. Test the ring after more concurrent writes than it can retain.

Define when expiry is observed, stage writes privately, and reject a conflicting commit. Use a small conservative protocol before optimizing concurrency.