TL;DR: Use a flexible 45-minute plan: five minutes for requirements, seven for the system shape, ten for sizing, sixteen for selected deep dives and seven to close. State assumptions and draw one request's path before adding machinery. Let the interviewer steer the depth. Finish with a concrete decision, its failure mode and the condition that changes it.
How to approach it
Offer the plan briefly, then ask for the facts that would change the design: workload, latency, scale, model, budget and constraints. Write non-goals. Explain assumptions when an input is missing and invite corrections. The schedule is a recovery aid, not something to impose on the interviewer.
A strong answer
Imagine reaching minute 30 with a detailed gateway drawing but no estimate of GPU demand. The useful correction is to stop expanding the drawing and show the missing capacity chain. The AI Infra Design Round Playbook helps organize that conversation; the example below illustrates a coding assistant, not a claim about any company's interview rubric.
In the opening five minutes, separate inline completions from chat if their models and latency contracts differ. Ask for arrival rates, prompt/output distributions and quality requirements; “40,000 online users” does not tell you how many requests arrive each second. State whether training, fine-tuning and multiple regions are outside this round's scope.
Next follow a request from the gateway to a replica that can serve it, through the engine scheduler and back to the client. Show where the engine allocates its key-value (KV) cache, the attention state retained while generating tokens. Name what each boundary owns: admission, cancellation, identity or durable state. Distinguish control-plane actions such as deployment from a request's data path. Trace one failure so the boxes already have operational meaning.
Use the sizing period for a complete but adjustable chain:
Hypothetical workload and measured-capacity assumption:
20,000 active users × 1 request per 20 seconds = 1,000 requests/s
fixed workload: 500 prompt tokens, 30 output tokens per request
input demand = 500,000 prompt tokens/s; output = 30,000 tokens/s
assume replay measured 50 qualifying requests/s per replica
target occupied fraction = 0.8
replicas = ceil(1,000 / (50 × 0.8)) = 25
illustrative cost $3 per replica-hour → $75/hour
Sanity: 25 × 50 × 0.8 = 1,000 requests/s; without headroom, 20 replicas.
That example assumes measured joint capacity on the same input/output mix and service objectives. It is not a GPU specification or a license to ignore memory. Validate model fit and per-rank cache, then replace the assumed replay capacity with evidence. If separate prefill and decode throughput figures came from isolated tests, do not treat both as simultaneously available on one shared pool.
Capacity Planning and Utilization makes sensitivity explicit. Doubling request frequency doubles the example's demand and, under unchanged per-replica performance, requires 50 replicas. Changing context length may also reduce per-replica capacity, so a simple twofold multiplier can then understate the fleet. Distinguish 20% unused capacity from adding 20% to a saturated fleet.
Offer three deep dives and ask which two matter most: routing under prefix affinity, admission under cache pressure, or autoscaling through startup delay. For a chosen topic, trace a concrete state change. For example, a router sends to a warm replica only while the saved prefill exceeds the added queue wait; the engine still performs final admission. A warm-cache preference is not permission to overload one host.
Reserve the last seven minutes for failure and rollout. Explain overload, lost replicas, cache-miss storms and cancellation; Google's overload guidance is a useful engineering reference for bounded work and graceful degradation. Describe what the canary measures, what triggers rollback and which data distinguishes a bad version from a changed workload.
Choose separate pools when their models or contracts require isolation, acknowledging stranded capacity and operational cost. Reverse that choice when a supported shared deployment meets both contracts more simply. Do not prescribe disaggregation merely because scale increases tenfold: identify the bottleneck and show which boundary would change. If budget tightens, revisit quality, model size, workload and objectives explicitly rather than silently weakening the contract.
What interviewers probe next
- "The round is already behind?" Summarize the current decision, show the missing sizing chain, and agree on one deep dive rather than rushing three.
- "Where did 50 requests/s come from?" It is a stated hypothetical measurement input; explain the replay that would establish it.
- "The budget halves?" Recompute options and name the product tradeoff. Do not pretend the same requirement costs half without a change.
Common mistakes
- Treating the minute plan as a rigid scoring rule.
- Using user count as request rate without an activity model.
- Drawing many services without tracing ownership and failure.
- Summing isolated capacity claims as though all resources were independent.
Key takeaways
- The durations 5, 7, 10, 16 and 7 sum to 45 minutes.
- The toy workload needs 25 replicas at 80% target occupancy.
- Make the main decision explicit and show what would change it.