Advanced

Batching Queues and Backpressure

Write a request batcher is the coding round's version of the serving engine's scheduler: requests arrive one at a time, the GPU wants them in groups, and the batcher decides when a group is full enough to send without holding anyone too long or accepting more than it can hold. The two knobs are the maximum batch size and the maximum wait, the invariant is a bounded queue, and the follow-ups (priorities, cost-aware batching, cancellation, bounded in-flight batches) are the ideas the real engines carry. This page implements the batcher in asyncio, derives what each knob buys, and walks the follow-ups.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Continuous Batching Capacity and Backpressure Producer-Consumer Pipelines Concurrency in Python, Go and C++Rate-Limiting Algorithms Designing for Latency SLOs

PRACTICE THIS IN REAL QUESTIONS

Coding for InfraWrite a producer-consumer pipeline with a bounded queue. What are the three bugs that show up in every first attempt?→Coding for InfraImplement a batcher that flushes when the batch is full or when a timeout expires. What breaks in the timer path?→Coding for InfraSimulate speculative decoding to find the expected tokens per round and the batch size where it stops paying.→AI Infrastructure System DesignDesign serving for a video generation model: diffusion steps, batching, memory, and a latency profile unlike an LLM's.→CUDA, Triton & Kernel EngineeringGiven the addresses each thread in a warp touched, classify the access pattern: coalesced, strided or random. Write the classifier.→Networking, Interconnects & StorageWhich NCCL environment variables are worth knowing, what does each change, and which should you almost never set?→