medium★ EssentialAnthropic

You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.

Preserve document identity through packed prompts and completion reordering. Derive rounded cache reservations, then measure feasible wave sizes against the API contract.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Preserve document identity through packed prompts and completion reordering. Derive rounded cache reservations, then measure feasible wave sizes against the API contract.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching changes which requests run between model iterations. Learn how a waiting request joins, why an empty slot is not enough, and how to balance throughput against each user’s token latency.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverMulti-region serving can improve geographic reach and survive a regional outage. It requires enough remaining capacity and a clear plan for state, routing and degraded service.

Foundational

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputRead a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

Foundational

📐 AI Systems Design

Control Plane and API Design for GPU PlatformsA control plane records intended work and coordinates its execution. Clear retry, cancellation and listing semantics keep its API useful when requests or workers fail.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.Next in this trackvLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?Next in this trackHow would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?

FEDITOR'S NOTE

Keep sequence boundaries and original request IDs through packing and sorting. Better device utilization is useful only if each result reaches the right caller.

DISCUSSION · 0

No comments yet — be the first to share your approach.