Core

Continuous Batching

Continuous batching changes which requests run between model iterations. Learn how a waiting request joins, why an empty slot is not enough, and how to balance throughput against each user’s token latency.

This concept is included with a free account. Sign in to continue reading.

free account · no card needed

RELATED CONCEPTS

Prefill vs Decode PagedAttention Chunked Prefill Latency Metrics: TTFT, TPOT and Goodput Batching Queues and Backpressure Capacity and Backpressure

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureA request becomes tokens, and tokens become two different workloads30 min→Foundations of AI InfrastructureWeight reuse explains why prefill and decode need different tuning19 min→Inference EngineeringFollow two requests through one loaded model21 min→

PRACTICE THIS IN REAL QUESTIONS

LLM Inference & ServingWhat is the difference between static and continuous batching, and why did it change LLM serving?→AI Infrastructure System DesignDesign an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.→Coding for InfraImplement the scheduler loop of a continuous-batching engine: admit, step, preempt. Where does it livelock?→Open-Weights Models & Serving EnginesThe engine logs say CUDA graphs are disabled for your deployment. Does it matter?→Open-Weights Models & Serving EnginesIs speculative decoding worth enabling on a trillion-parameter mixture-of-experts model?→LLM Inference & ServingYou have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.→