TL;DR: Minutes 0 to 5, pin the requirements as numbers and write the non-goals. Minutes 5 to 12, draw the reference shape for the family, one job per box. Minutes 12 to 22, the sizing chain out loud: demand, capacity per unit, units, headroom, cost, with a sanity check. Minutes 22 to 38, two deep dives from a menu of three you offer. Minutes 38 to 45, failure modes, rollout, observability and what changes at 10×. Commit to the main trade-off and say what would reverse it.
How to approach it
Say the plan before executing it, because the interviewer grades the structure as much as the content: "I will confirm the numbers, draw the standard shape, size it, then go deep where you want." Ask for users, model, latency targets, scale in a year and budget, and write them on the board. Name what is out of scope so the round stays inside the hour. Then drive: the candidate who waits to be asked for each step is graded on the interviewer's structure instead of their own.
A strong answer
A typical situation: a candidate draws for twenty minutes, reaches the sizing with eight left, and the interviewer never finds out whether they can do the arithmetic. The plan below exists to notice that at minute 12 rather than at minute 40.
The AI Infra Design Round Playbook is the same for every prompt; only the shape and the chain change. The version below runs it against a concrete prompt: the serving platform for a coding assistant.
Minutes 0 to 5: the numbers. For the coding assistant: 200k daily users, 40k concurrent at peak; two workloads, inline completions (about 500 tokens of context, 30 output tokens, TTFT p95 under 200 ms) and chat (about 4k context, 400 output, TTFT p95 under 700 ms, TPOT p95 under 60 ms); a 7B for completions and a 70B for chat, both self-hosted; 3× growth in a year; non-goals: training, fine-tuning, multi-region. The decision already falls out before a box is drawn: two pools, because the two workloads have different latency shapes. Say that sentence at minute five.
Minutes 5 to 12: the shape. Gateway, router, per-pool replicas with an engine scheduler inside, KV tier, autoscaler, observability. One sentence per box, with the failure it owns. Do not describe the engine as a black box; the scheduler and the KV pool are where the follow-ups go.
Minutes 12 to 22: the chain. Every number is an assumption the interviewer can change, so say each one.
completions pool (7B, one H100 per replica, fp8 weights 7 GB)
demand: 40k concurrent × 1 completion per 10 s = 4,000 req/s × 500 tokens = 2 M prefill tok/s
prefill per GPU at fp8: 1,979e12 × 0.4 MFU ÷ (2 × 7e9 FLOPs per token) ≈ 57,000 tok/s
replicas: 2 M ÷ 57,000 ≈ 35; +20% headroom → 42 H100s
TTFT check: 500 tokens × 2 × 7e9 ÷ (1,979e12 × 0.4) ≈ 9 ms of compute, so the 200 ms budget is queueing
chat pool (70B, 8 × H100 per replica)
mid-generation fraction ≈ 400 × 60 ms = 24 s ÷ (24 + 60 s reading) ≈ 30%
decode demand: say 8k of the 40k are in chat: 8,000 × 0.3 × 17 tok/s ≈ 41,000 tok/s
goodput per replica at p95 TPOT 60 ms: take 2,200 tok/s; prefill takes 30% of each replica
replicas: 41,000 ÷ 2,200 ÷ 0.7 ≈ 27; +20% → 32 replicas = 256 H100s
cost: (42 + 256) × $2.50/h ≈ $745/h at peak; sanity: about $0.02 per user-hour at 40k concurrent
Say the sensitivity: the chat pool is 85% of the bill and its size is set by the TPOT target and the prefix hit rate, so those are the two inputs to negotiate.
Minutes 22 to 38: the deep dives. Offer three: the router's prefix affinity, the engine scheduler's admission, the autoscaler's cold-start handling. For whichever two are chosen, explain the mechanism, not the name. Prefix affinity: hash the conversation to a replica, correct with load, and handle the restart miss storm. Admission: a bounded queue whose depth equals the wait budget times the service rate, and a KV check on the request's maximum pages before admission. Cold start: a warm pool sized to the largest daily step, scaling on pending tokens rather than GPU utilization.
Minutes 38 to 45: the close. Failure modes without being asked: full KV pool, lost affinity on rollout, a slow replica hidden by least-loaded routing, a traffic step faster than a cold start. Rollout: canary a few replicas per pool and compare TTFT and quality per version. At 10×: the pools become hundreds of nodes, prefill and decode disaggregate, and prefix reuse moves to a shared tier.
The trade-off to commit to here is two pools against one. Two pools cost some utilization (each pool carries its own headroom) and buy separate SLOs. The reversal condition: if the completion traffic falls under a few hundred requests per second, a single pool with priority lanes carries both without the double headroom. The AI Infra Design Round Playbook is the longer form of this plan, and Capacity Planning and Utilization supplies most of the numbers minute 12 needs.
What interviewers probe next
- "You spent ten minutes on arithmetic; why?" Because without the chain the drawing does not say whether to order 40 GPUs or 4,000, and every follow-up is a change to one of its inputs.
- "What if I told you the budget is a third of that?" Cut the chat model to a smaller one or relax TPOT to 100 ms, and show the chain again with the new numbers; do not cut headroom.
- "What would you do differently at a startup with one team?" Same structure, smaller shape: one pool, one router, no disaggregation, and the non-goals list gets longer.
Common mistakes
- Spending fifteen minutes on requirements and drawing nothing.
- A diagram with no numbers attached to any box.
- "It depends" with no statement of what it depends on and which way you would go.
- Solving for the wrong scale: a three-replica design for 40k concurrent users, or a global control plane for one team.
- Ignoring the hardware: a serving design that never mentions memory bandwidth is a web-services design with the nouns changed.
Key takeaways
- 5, 7, 10, 16, 5: the minute plan in minutes, and say it out loud at the start so the interviewer can steer it.
- Requirements as numbers with non-goals; a shape with one job per box; a chain with every assumption stated and a sanity check.
- Offer three deep dives, know the mechanism in each, and name failures before being asked.
- Commit to the trade-off and state the reversal condition.