continuous batching

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

From an admission queue to a streamed token, the batching system that decides how many tokens per second a replica earns and what its p95 TPOT is. The decode arithmetic that sets the batch, the KV budget that caps it, and the streaming path that must never stall the engine.

Admission, a step that grows every sequence, and preemption under memory pressure. The loop is short and it has a failure that only appears when the cache is tight: in a measured run it preempted seventeen times and completed nothing, because every preempted sequence restarted from zero.