AI Infrastructure System Design Interview Questions

Seven boxes between an API call and a GPU, each with one job and one way to fail. The walkthrough a screen expects in the first ten minutes, with the sizing chain from 2,000 concurrent users to a replica count so the drawing has numbers on it.

Five minutes of numbers, seven of shape, ten of arithmetic, sixteen of depth, five of failure modes. The minute plan that keeps a serving-platform design inside 45 minutes, with the worked opening for a coding assistant and the mistakes that end the round early.

From an admission queue to a streamed token, the batching system that decides how many tokens per second a replica earns and what its p95 TPOT is. The decode arithmetic that sets the batch, the KV budget that caps it, and the streaming path that must never stall the engine.

A GPU credit system is an accounting ledger with a scheduler attached: who may spend, how fast, what happens when the account is empty and what happens when everyone spends at once. The data model, the debit path, the fairness rule and the arithmetic that keeps 4,096 GPUs busy without starving a team.

Ten thousand queries a second through an embedder, a sharded vector index, a reranker and a 70B answer model. The latency budget per stage, the two fleets sized from tokens and pairs rather than queries, and the cache that decides whether the LLM tier is 60 nodes or 200.

Ten thousand GPUs is 1,280 nodes, ten pods, about 700 switches, a filesystem that swallows a 5 TB checkpoint in under a minute and a 14 MW feed. The bill of materials with a reason on every line, from the FLOPs the model needs to the spare pool the failure rate demands.

A hundred thousand jobs is a queue you cannot scan per tick, a preemption policy that has to know when each victim last checkpointed, and a state machine with one transition that most designs get wrong. The data model, the tick loop, the checkpoint-aware eviction and the arithmetic behind each.

A serverless GPU is a promise to run a customer's model within seconds of a request while charging only for the seconds it runs. The cold-start chain from image pull to first token, the snapshot that collapses it, the warm pool sized by Little's law, and the isolation and packing decisions that set the margin.

Two thousand evals against every checkpoint is ten million prompts per run and the difference between a four-hour and a two-day turnaround. The pipeline as a batch inference job with a content-addressed cache, the sharding that keeps GPUs busy, and the reproducibility rules that let a regression be believed.

Fifteen trillion tokens starts as a few petabytes of raw text and ends as 30 TB of shards a training job reads at 1.7 million tokens per second. The stages, the bytes at each boundary, the throughput each one needs to finish in two weeks, and the two stages where the pipeline actually spends its time.

A replica per variant is 100 copies of a 140 GB base and 800 GPUs mostly idle; multi-LoRA holds the base once and hundreds of adapters in a few gigabytes. The memory arithmetic, the throughput cost of mixed batches, the routing that keeps hot adapters resident, and the two cases where a variant earns its own replica.

A replica takes 70 seconds to become useful and the burst arrives in 60, so the autoscaler cannot win the first minute; the warm pool has to. The leading signal, the cooldowns derived from the cold start, the warm-pool size from Little's law and the burst profile, and the cost of every second of cold start left in.

A 70B model's KV cache is 320 KB per token, and a 30k-token conversation is 10 GB that GPU memory cannot keep between turns. The three tiers with their bandwidths, the break-even where reloading a prefix beats recomputing it, the eviction and lookup design, and the traffic shape where the tier is worth its complexity.

A 141 GB artifact is not a Docker image, and shipping it like one costs an hour per node and a registry that melts under a rollout. Content-addressed chunks, a manifest per version, a fan-out that turns one origin into a thousand nodes in minutes, integrity that fails closed, and promotion tied to evaluation.

A request can cost 50 tokens or 50,000, so a request limit protects nothing. The two buckets per tenant per model, the reservation-then-settle scheme for output tokens you cannot count in advance, the distributed counter fast enough for the gateway, and how the limits map onto the fleet's real capacity.

Two regions at 60% is not one region with a spare: the question is what happens in the 30 seconds after a region drops. Capacity sized for N-1 with the arithmetic, latency routing with a residency override, failover that does not stampede, weights warm everywhere in advance, and the state that must not cross a border.

The playground is the API's most public client and its most abused one: free, barely identified, streaming. The transport and its cost per connection, session state that survives a refresh with no GPU holding anything, quotas per user and per fleet, and the abuse controls that stop the free tier becoming a free API.

Every 30 minutes a thousand GPUs write a terabyte in a burst that must finish in a minute, then nothing until the next. The burst arithmetic that sizes the write tier, the two-stage path to durable storage, retention that keeps the right checkpoints, resharding so a 512-GPU checkpoint resumes on 256, and lineage.

GPU utilization at 100% tells you nothing, and a trace per token would cost more than the tokens. The span layout for one request, the GPU counters that read headroom, the cardinality budget that keeps the metrics store alive, and cost attribution that charges tenants for the batch share they used.

A GPU cloud sells whole nodes and fabric slices to tenants who expect them in minutes and expect them to work. The node state machine, the reconciler that keeps thousands converging, the isolation lines on shared fabric, the health loop that pulls a bad node first, and billing that survives a control-plane outage.

200 milliseconds at p99 is a budget, and prefill alone spends it on a long prompt. The decomposition into network, queue, routing, prefill and first token, the prompt length the budget allows, chunked prefill in the tail, admission control that rejects what it cannot serve in time, and the utilization the SLO costs.

An agent thinks for a second, runs a tool for a minute, and repeats a hundred times over an hour. The session state machine and where each piece lives, the sandbox per session with its cold-start budget, the KV cache that should survive the tool call, and checkpoints that resume a step rather than an hour.

A video request is not a stream of tokens: it is 40 denoising passes over a latent the size of a small film, and the user waits a minute for the whole thing. The memory and FLOP arithmetic per request, why batching works differently, the queue and progress design for minute-long jobs, and where the cost goes.

Speculative decoding buys latency with spare compute, and a busy fleet has no spare compute. The speedup formula with acceptance rate and draft cost inside it, the batch size where the gain turns negative, where the draft runs and what it costs, the monitoring that catches a silent regression, and rollout per class.

A research cluster is shared by teams whose managers bought it and by researchers who want it now. A quota tree with guarantees and borrowing, a fair-share formula that decays so last week's usage does not cost you today, preemption that is cheap because checkpoints are, and the incentives that make hoarding lose.

A three-month run on 8,000 GPUs is a large bet against everything that can happen to one building. The recovery point and the recovery time as numbers, the checkpoint replication that sets the first, the second cluster and its warm state that set the second, and the drills that make the numbers true.

Billing is the one pipeline where a dropped event is money and a duplicated one is a support ticket. Where tokens are counted and by whom, the usage event and its idempotency key, the aggregation that closes a month, the reconciliation that proves the ledger against the engines, and the six ways it goes wrong.

Prefill is compute-bound and decode is bandwidth-bound, and one GPU running both does each worse. The pool ratio from the traffic's prompt and output lengths, the KV transfer cost per request and the link it needs, the router that pairs a prefill worker with a decode worker, and the load at which it beats a monolith.

Forty teams calling three providers with a hundred API keys and no idea what it costs is what the gateway exists to end. One endpoint with a model alias layer, routing by policy and health, a cache that is safe only for deterministic calls, fallbacks that degrade rather than fail, budgets per team, and the audit trail.

A trillion parameters with 32 billion active per token: the weights need a rack, the compute needs a fraction of it, and the all-to-all between them decides everything. The memory plan across 72 GPUs, the decode step from bandwidth and two all-to-alls, the compressed attention cache, and what imbalance costs.

The first question is whether 1T is total or active, because storage follows one and compute follows the other. The state budget, the layout that falls out of it, why expert parallelism belongs inside NVLink, and the two failure modes a dense-model plan does not have: router imbalance and all-to-all congestion.