← AI Infrastructure System Design / 33

hardNew

Review this eight-GPU inference design: what must change before it can meet its deadline?

Find concrete failure paths in a small serving architecture. Rank correctness, admission and placement fixes before proposing additional infrastructure.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Find concrete failure paths in a small serving architecture. Rank correctness, admission and placement fixes before proposing additional infrastructure.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 303 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🚀 Inference & Serving🔒 Premium

Structured Outputs and Constrained DecodingConstrained decoding rules out tokens that break a grammar. Follow a two-choice example through masking, completion and rollback, then separate valid structure from correct, authorized tool arguments.

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

📐 AI Systems Design🔒 Premium

Capacity and BackpressureBackpressure limits accepted work when a service cannot keep up. Bounded queues and clear retry rules prevent overload from spreading through the request path.

📐 AI Systems DesignSign in

GPU Job Scheduler DesignA GPU scheduler decides which jobs run, where they run and when resources can be reclaimed. Its placement rules must account for whole jobs, topology and tenant policy.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign an online and offline feature store without leaking future information into training Closest to this topicDesign an LLM batching system end to end: the queue, the batch, the KV cache and streaming. Give me numbers.Closest to this topicOne fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.

DISCUSSION · 0

No comments yet — be the first to share your approach.