hardNewGoogleOpenAIFireworks

Design serving for a video generation model: diffusion steps, batching, memory, and a latency profile unlike an LLM's.

A video request is not a stream of tokens: it is 40 denoising passes over a latent the size of a small film, and the user waits a minute for the whole thing. The memory and FLOP arithmetic per request, why batching works differently, the queue and progress design for minute-long jobs, and where the cost goes.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

💻 Coding for Infra🔒 Premium

Batching Queues and BackpressureWrite a request batcher is the coding round's version of the serving engine's scheduler: requests arrive one at a time, the GPU wants them in groups, and the batcher decides when a group is full enough to send without holding anyone too long or accepting more than it can hold. The two knobs are the maximum batch size and the maximum wait, the invariant is a bounded queue, and the follow-ups (priorities, cost-aware batching, cancellation, bounded in-flight batches) are the ideas the real engines carry. This page implements the batcher in asyncio, derives what each knob buys, and walks the follow-ups.

Core

📐 AI Systems DesignSign in

GPU Job Scheduler DesignDesign a scheduler for a shared GPU cluster is the most common design prompt in AI infrastructure interviews, because it touches everything: queues and priorities, gang placement, topology, fairness across teams, preemption and the checkpoints that make it survivable, and the failure handling that keeps a 512-GPU job alive. This page builds the design in layers, states the data model and the scheduling loop, derives the numbers (how long a job waits, how much preemption costs, how much fragmentation wastes), and lists the trade-offs the interviewer will push on.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Gang Scheduling with Kueue and VolcanoA distributed training job is 64 pods that start together or not at all: if 40 are running and 24 are Pending, the 40 hold their GPUs idle at a collective barrier waiting for ranks that may never come, and two such jobs can deadlock a whole cluster. Gang scheduling makes the job the unit of admission. Kueue and Volcano add queues, quotas, priorities and preemption on top, which is what turns a pile of GPUs into a platform several teams can share without starving each other.

Core

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign speculative decoding into a production serving fleet: draft placement, acceptance monitoring, and the batch regime where it pays.Next in this trackDesign quota and fairness for a shared research cluster: hierarchical quotas, preemption, and the incentives that keep it honest.Next in this trackDesign disaster recovery for a three-month training run: checkpoint replication, cluster failover, and the RTO you can promise.

FEDITOR'S NOTE

Scored on the per-request arithmetic (latent size, FLOPs per step, steps, seconds on a node), on treating requests as asynchronous jobs with progress rather than streams, on batching by shape and step-level sharing, and on the cost per clip as the number the design is judged by.

DISCUSSION · 0

No comments yet — be the first to share your approach.