← AI Infrastructure System Design / 32

hardNew

Design an asynchronous batch inference API for a billion records with a completion deadline

Connect POST-and-poll semantics to durable item results, retry ownership and deadline capacity. Explain how cheap batch execution can still miss the delivery contract.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Connect POST-and-poll semantics to durable item results, retry ownership and deadline capacity. Explain how cheap batch execution can still miss the delivery contract.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 303 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureData pipelines prepare traceable training inputs; evaluation pipelines compare model behavior. Both need versioned inputs, reproducible execution and useful failure reporting.

📐 AI Systems Design

Control Plane and API Design for GPU PlatformsA control plane records intended work and coordinates its execution. Clear retry, cancellation and listing semantics keep its API useful when requests or workers fail.

💻 Coding for InfraSign in

Retry, Backoff and IdempotencyRetries can recover transient failures, but they also add load and may repeat side effects. Backoff, retry budgets and idempotency make that behavior explicit.

🧮 Napkin Math & Capacity🔒 Premium

Capacity Planning and UtilizationCapacity planning connects demand, useful throughput and acceptable waiting time. Serving and training need different models of bursts, job shape and spare capacity.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackReview this eight-GPU inference design: what must change before it can meet its deadline?Next in this trackDesign an online and offline feature store without leaking future information into training Closest to this topicDesign the API for a GPU job scheduler in 45 minutes. What are the resources, states and semantics?

DISCUSSION · 0

No comments yet — be the first to share your approach.