The AI Infra Design Round Playbook

TL;DR: Clarify the workload and constraints, draw a system that meets them, and check its size with explicit assumptions. Choose a few important mechanisms to explain in depth. Leave time for failures, rollout and cost, adapting the pace to the interviewer rather than following a fixed minute-by-minute script.

The minute-by-minute plan

Adapt the plan to the prompt and discussion. The table is one rehearsal schedule, not a required sequence. The pages for each family carry them: Inference Platform Architecture, GPU Job Scheduler Design, Training Cluster Design at 10k GPUs, Multi-Tenant Fine-Tuning Service, Evaluation and Data Pipeline Infrastructure.

The numbers to bring

An AI infra design round is a napkin-math round with a whiteboard. These are the ones to have without looking:

memory
  weights = params × bytes/param: bf16 2, fp8 1, int4 0.5  → 70B bf16 = 140 GB; 405B bf16 = 810 GB
  KV per token = 2 × layers × kv_heads × head_dim × bytes → Llama 3 70B bf16: 2 × 80 × 8 × 128 × 2 = 320 KB
  training state ≈ 16 bytes/param (weights, grads, Adam in mixed precision) → 70B ≈ 1.1 TB before activations
compute
  training FLOPs ≈ 6 × params × tokens → 70B on 15T tokens ≈ 6.3 × 10²⁴
  time = FLOPs ÷ (GPUs × peak × MFU) → at 8,192 H100s, 990 TFLOPS dense bf16, 40% MFU: ≈ 22 days
  per-GPU: H100 ~990 TFLOPS bf16 dense, 3.35 TB/s HBM; B200 ~2.2 PFLOPS bf16 dense, 8 TB/s; ridge ≈ 295 FLOP/byte (H100)
serving
  decode is bandwidth-bound: tok/s per replica ≈ batch × BW ÷ (weights + batch × KV per sequence)
  TTFT ≈ prefill tokens × 2 × params ÷ (peak × MFU): 2,000 tokens on 70B at 8 H100s, 40% MFU ≈ 90 ms compute
  chat SLOs: p95 TTFT 500 ms, p95 TPOT 50 ms; reading speed ≈ 5 tok/s
cost and reliability
  H100 ≈ $2 to 3 per GPU-hour rented; cost per million tokens = ($/h ÷ 3600) ÷ (tok/s × utilization) × 10⁶
  failure rate ≈ 2 × 10⁻⁵ per GPU-hour → a stop every 3 h at 16k GPUs; loss per stop ≈ T/2 + R
  network: 400 Gb/s per NIC ≈ 50 GB/s; NVLink 900 GB/s (H100), 1.8 TB/s (B200); ring all-reduce moves 2(N−1)/N × size per GPU

The full set with derivations is on the formulas page and in the napkin-math track.

What the interviewer is scoring

• Requirements discipline: did you ask for the numbers, and did you write non-goals?
• Shape: do you know the standard architecture for this family and what each box does?
• Arithmetic: can you get from demand to a bill of materials with stated assumptions and a sanity check?
• Depth: in the deep dive, do you know the mechanism (how prefix-aware routing decides, how gang admission avoids deadlock) rather than the name?
• Judgment: do you name trade-offs and commit to one, with the condition under which you would reverse?
• Operability: failure modes, rollout, observability, cost, without prompting.

Leveling shows up in judgment and operability: broader ownership includes anticipating failure modes and connecting them to service goals; level expectations vary by employer (Leveling Signals: Senior vs Staff).

Common problems to practice avoiding

1. No numbers. A design without a sizing chain is a diagram, and the interviewer cannot tell whether you would order 40 GPUs or 4,000.
2. The opaque box. "GPU workers" with nothing inside. The engine's scheduler, the KV pool and the batch are where the design lives.
3. Wrong scale. Designing a three-replica service for a prompt that said a million users, or a global control plane for a prompt that said one team.
4. Refusing the trade-off. "It depends" without saying on what and which way you would go. Commit, and state the reversal condition.
5. Ignoring the hardware. A serving design that never mentions memory bandwidth, or a training design that never mentions the fabric, is a web-services design with the nouns changed.
6. Running out of time in the requirements. Five minutes, then draw.
7. Not driving. The candidate who waits to be asked for each next step is graded on the interviewer's structure, not their own.

A worked opening

prompt: "Design the serving platform for our coding assistant."
minute 0 to 5, the questions and the numbers written on the board:
  users: 2M monthly, 200k daily, peak 40k concurrent
  workload: inline completions (short prompts ~500 tokens of context, ~30 output tokens, TTFT p95 ≤ 200 ms)
            and chat (~4k context, ~400 output, TTFT p95 ≤ 700 ms, TPOT p95 ≤ 60 ms)
  models: a 7B for completions, a 70B for chat; open weights, we host them
  scale in a year: 3×;  budget: "reasonable";  non-goals: training, fine-tuning, multi-region for now
minute 5: "Two pools, one per model, behind one gateway and router; the completion pool is TTFT-dominated
          and the chat pool is TPOT-dominated, so I'll size them differently. Let me draw it."

That opening already contains the decision (two pools) and the reason (two latency shapes), before a single box is drawn.

Working it in the room

Practice with a timer to see where explanations become too long or skip necessary steps. The suggested timings are a rehearsal aid. During the interview, respond to the discussion: summarize completed decisions, check assumptions and spend time on the parts that need more reasoning. A useful answer makes the design understandable and open to challenge.

What to remember

• Five minutes of requirements and numbers, seven of shape, ten of sizing, sixteen of deep dives, five of failure modes and cost.
• Carry the numbers: bytes per param, KV per token, 6ND, peak and bandwidth for two parts, the decode formula, SLO thresholds, cost per GPU-hour, failure rate, checkpoint arithmetic.
• Offer three deep dives, know the mechanism in each, commit to trade-offs with a reversal condition.
• Name failure modes, rollout and observability before being asked.
• Practice explaining scale, mechanisms and tradeoffs clearly enough that another person can challenge the design.