The AI Infra Design Round Playbook
The AI infrastructure design round is 45 to 60 minutes with one prompt (design a serving platform, a training scheduler, a 10k-GPU cluster, a fine-tuning service) and one interviewer whose job is to find the edge of what you know. The candidates who pass do the same things in the same order: pin the requirements and the numbers in the first five minutes, draw the reference shape, size it with a stated chain of arithmetic, pick two deep dives, and name the failure modes before being asked. This page gives that structure with a minute-by-minute plan, the numbers to bring in your head, and the mistakes that end the round early.
TL;DR: Minutes 0 to 5: requirements and numbers (users, tokens, model size, SLOs, scale, budget, non-goals), written down. Minutes 5 to 12: the reference shape on the board, every box with one job. Minutes 12 to 22: the sizing chain, out loud, with each assumption stated and a sanity check at the end. Minutes 22 to 38: two deep dives the interviewer chooses from the three you offer. Minutes 38 to 45: failure modes, rollout, cost, what you would change at 10× scale. The numbers to carry: bytes per parameter by dtype, KV per token, the 6ND rule, per-GPU peak and bandwidth for two or three parts, the ridge point, a decode throughput formula, cost per GPU-hour, failure rate per GPU, checkpoint arithmetic. The mistakes that end it: no numbers, a diagram with a "GPU worker" box and nothing inside, solving the wrong scale, and refusing to commit to a trade-off.
The minute-by-minute plan
| Minutes | Do | Say out loud |
|---|---|---|
| 0 to 5 | pin the requirements: who calls it, how many, what model, what latency, what scale in a year, what budget, what is out of scope | "Let me confirm the numbers before drawing anything" |
| 5 to 12 | draw the reference shape for the prompt's family (serving, scheduler, cluster, pipeline), one job per box | "I'll draw the standard shape first, then we'll size it and go deep where you want" |
| 12 to 22 | the sizing chain: demand → per-unit capacity → units → headroom → cost; state every assumption; sanity-check the total | "Here is how I get from 10,000 users to 40 replicas, and which assumption moves it most" |
| 22 to 38 | two deep dives, offered as a menu of three (the router, the scheduler's admission, the KV tier; or placement, preemption, failure handling) | "Which of these would you like to go into?" |
| 38 to 45 | failure modes, rollout and rollback, observability, what changes at 10× | "Before we finish, the ways this breaks and what I'd watch" |
The plan is the same for every prompt; only the reference shape and the sizing chain change. The pages for each family carry them: Inference Platform Architecture, GPU Job Scheduler Design, Training Cluster Design at 10k GPUs, Multi-Tenant Fine-Tuning Service, Evaluation and Data Pipeline Infrastructure.
The numbers to bring
An AI infra design round is a napkin-math round with a whiteboard. These are the ones to have without looking:
memory
weights = params × bytes/param: bf16 2, fp8 1, int4 0.5 → 70B bf16 = 140 GB; 405B bf16 = 810 GB
KV per token = 2 × layers × kv_heads × head_dim × bytes → Llama 3 70B bf16: 2 × 80 × 8 × 128 × 2 = 320 KB
training state ≈ 16 bytes/param (weights, grads, Adam in mixed precision) → 70B ≈ 1.1 TB before activations
compute
training FLOPs ≈ 6 × params × tokens → 70B on 15T tokens ≈ 6.3 × 10²⁴
time = FLOPs ÷ (GPUs × peak × MFU) → at 8,192 H100s, 990 TFLOPS dense bf16, 40% MFU: ≈ 22 days
per-GPU: H100 ~990 TFLOPS bf16 dense, 3.35 TB/s HBM; B200 ~2.2 PFLOPS bf16 dense, 8 TB/s; ridge ≈ 295 FLOP/byte (H100)
serving
decode is bandwidth-bound: tok/s per replica ≈ batch × BW ÷ (weights + batch × KV per sequence)
TTFT ≈ prefill tokens × 2 × params ÷ (peak × MFU): 2,000 tokens on 70B at 8 H100s, 40% MFU ≈ 90 ms compute
chat SLOs: p95 TTFT 500 ms, p95 TPOT 50 ms; reading speed ≈ 5 tok/s
cost and reliability
H100 ≈ $2 to 3 per GPU-hour rented; cost per million tokens = ($/h ÷ 3600) ÷ (tok/s × utilization) × 10⁶
failure rate ≈ 2 × 10⁻⁵ per GPU-hour → a stop every 3 h at 16k GPUs; loss per stop ≈ T/2 + R
network: 400 Gb/s per NIC ≈ 50 GB/s; NVLink 900 GB/s (H100), 1.8 TB/s (B200); ring all-reduce moves 2(N−1)/N × size per GPU
The full set with derivations is on the formulas page and in the napkin-math track.
What the interviewer is scoring
- Requirements discipline: did you ask for the numbers, and did you write non-goals?
- Shape: do you know the standard architecture for this family and what each box does?
- Arithmetic: can you get from demand to a bill of materials with stated assumptions and a sanity check?
- Depth: in the deep dive, do you know the mechanism (how prefix-aware routing decides, how gang admission avoids deadlock) rather than the name?
- Judgment: do you name trade-offs and commit to one, with the condition under which you would reverse?
- Operability: failure modes, rollout, observability, cost, without prompting.
Leveling shows up in judgment and operability: a senior candidate names the failure modes when asked; a staff candidate names them before being asked and connects them to the SLO (Leveling Signals: Senior vs Staff).
The mistakes that end the round
- No numbers. A design without a sizing chain is a diagram, and the interviewer cannot tell whether you would order 40 GPUs or 4,000.
- The opaque box. "GPU workers" with nothing inside. The engine's scheduler, the KV pool and the batch are where the design lives.
- Wrong scale. Designing a three-replica service for a prompt that said a million users, or a global control plane for a prompt that said one team.
- Refusing the trade-off. "It depends" without saying on what and which way you would go. Commit, and state the reversal condition.
- Ignoring the hardware. A serving design that never mentions memory bandwidth, or a training design that never mentions the fabric, is a web-services design with the nouns changed.
- Running out of time in the requirements. Five minutes, then draw.
- Not driving. The candidate who waits to be asked for each next step is graded on the interviewer's structure, not their own.
A worked opening
prompt: "Design the serving platform for our coding assistant."
minute 0 to 5, the questions and the numbers written on the board:
users: 2M monthly, 200k daily, peak 40k concurrent
workload: inline completions (short prompts ~500 tokens of context, ~30 output tokens, TTFT p95 ≤ 200 ms)
and chat (~4k context, ~400 output, TTFT p95 ≤ 700 ms, TPOT p95 ≤ 60 ms)
models: a 7B for completions, a 70B for chat; open weights, we host them
scale in a year: 3×; budget: "reasonable"; non-goals: training, fine-tuning, multi-region for now
minute 5: "Two pools, one per model, behind one gateway and router; the completion pool is TTFT-dominated
and the chat pool is TPOT-dominated, so I'll size them differently. Let me draw it."
That opening already contains the decision (two pools) and the reason (two latency shapes), before a single box is drawn.
Working it in the room
The playbook is the answer to "how should I approach this?", and the practice is running it against each family's page with a timer: five minutes of requirements, seven of shape, ten of arithmetic, then two deep dives with the mechanism explained, then the closing five. Record yourself; the mistakes on the list are audible.
What to remember
- Five minutes of requirements and numbers, seven of shape, ten of sizing, sixteen of deep dives, five of failure modes and cost.
- Carry the numbers: bytes per param, KV per token, 6ND, peak and bandwidth for two parts, the decode formula, SLO thresholds, cost per GPU-hour, failure rate, checkpoint arithmetic.
- Offer three deep dives, know the mechanism in each, commit to trade-offs with a reversal condition.
- Name failure modes, rollout and observability before being asked.
- The round ends early on no numbers, an opaque GPU box, the wrong scale, or an unwillingness to decide.