backpressure
AI infra interview questions tagged backpressure, across every topic.
2 questions · 1 unlocked for you
Concepts behind "backpressure"
The curriculum that explains the ideas these questions test.
Advanced
Capacity and BackpressureA system that accepts more work than it can finish does not degrade gracefully; it degrades completely, because every request it queues makes every other request slower until all of them time out. The defence is backpressure: bounded queues at every stage, admission control that rejects early when the expected wait exceeds the budget, load shedding by priority, and clients that back off. For LLM serving the stages are the gateway, the router, the engine's queue and its KV pool. This page works the arithmetic of why unbounded queues kill p99, designs the bounds per stage, and the client contract that keeps it stable under overload.📐 AI Systems Design🔒 Premium
Advanced
Batching Queues and BackpressureWrite a request batcher is the coding round's version of the serving engine's scheduler: requests arrive one at a time, the GPU wants them in groups, and the batcher decides when a group is full enough to send without holding anyone too long or accepting more than it can hold. The two knobs are the maximum batch size and the maximum wait, the invariant is a bounded queue, and the follow-ups (priorities, cost-aware batching, cancellation, bounded in-flight batches) are the ideas the real engines carry. This page implements the batcher in asyncio, derives what each knob buys, and walks the follow-ups.💻 Coding for Infra🔒 Premium
Advanced
Producer-Consumer PipelinesA data loader, a log shipper, a batch inference job and a checkpoint writer are the same program: stages connected by bounded buffers, each running at its own pace, the slowest setting the throughput and the buffers absorbing the jitter between them. The coding screen asks you to build one (read, decode, batch, feed a consumer) and then pushes on the production questions: buffer sizes, clean stops, failure propagation, and why it runs at a third of the expected speed. This page derives throughput from stage times, implements the pipeline in threads and asyncio, and works the stop and failure semantics.💻 Coding for Infra🔒 Premium