AI Infrastructure System Design Interview Questions

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Use a flexible 45-minute plan to connect requirements, a request path, capacity arithmetic, selected deep dives and a clear closing decision.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.

Trace GPU credits through reservation, allocation, metering and settlement, with crash-safe ownership and a separate fairness policy.

Follow cache hits and misses into shard calls, rerank pairs and generation work; size consistent branches and enforce access-aware reuse.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

Eight ready slots meet a twelve-request burst. See why starting instances do not count as ready capacity, then design caches, isolation, admission and a measurable startup contract.

Four result rows can still hide a missing case. Build immutable evaluation identities, retry-safe commits, versioned scoring and a capacity estimate tied to measured prompt throughput.

A 128,000-ID vocabulary breaks the 16-bit storage estimate. Follow the bytes from raw ingestion to 60 TB of token payload, then design deduplication, manifests and a restartable loader.

Derive adapter bytes from the target matrices, share compatible base weights, and keep each request's cache identity separate. A worked pool design avoids applying the mixed-adapter cost twice.

A 60-second demand ramp meets a 70-second startup delay. Compare the rate and ready-capacity curves, then design a controller that tracks pending launches without a blind cooldown.

A reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.

Bind weights, tokenizer, configuration and evaluation to one immutable release. Follow verification, prefetch, canary readiness and rollback, with bandwidth floors that do not pretend to be rollout measurements.

Divide one token budget across gateways instead of copying it. Follow output reservation and settlement, then handle grants, retries and outages without inventing extra burst capacity.

Remove the largest region from a concrete capacity plan. Check the surviving budget, tenant eligibility, cold prefixes and retry semantics before claiming the service can fail over.

Separate durable conversation state from live generation, bound streaming buffers, and make reconnect, cancellation and free-tier budgets explicit.

Derive checkpoint bytes, burst and drain bandwidth; publish only complete manifests, then prove restore and retention work under failures.

Trace a stalled stream without unbounded telemetry, understand what sampling loses, and reconcile GPU allocation with a stated cost policy.

Follow GPU ownership from reservation through verified delivery and cleanup, with stale-operation fencing, recovery and auditable metering.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.

Separate session state, model calls and tool execution. Calculate growing context, budget isolated workspaces and recover uncertain actions without duplicating side effects.

Follow a video job from prompt to downloadable artifact. Work the latent-shape arithmetic, compare batching with actual delivery times, and design bounded queues, cancellation and safe retries.

Compare the output of a speculative round with drafting and verification cost. Work the break-even condition, then design a measured rollout with memory and correctness checks.

A team returns to claim its GPUs, but another team is borrowing them in one indivisible job. Work through the quota tree, whole-job reclaim, recovery costs and release-to-ready budget before promising a start time.

Derive checkpoint replication and recovery budgets, identify the state a second cluster needs, and test the difference between a copied artifact and resumed training.

Trace usage from execution to one settled charge. Distinguish requests, attempts and events, then handle cancellation, late records, replay and reconciliation.

Size both serving pools from workload-matched rates, then check memory, shared transfer paths and surviving capacity. Follow admission and buffer ownership through failure and scaling.

Define model aliases, permission-aware routing, response reuse and bounded retries. Trace a request whose cached answer becomes invalid after its source permissions change.

Size a hypothetical trillion-parameter MoE from its weight placement, attention-state layout and expert traffic. Use rack specifications as bounds, then measure the serving workload before promising throughput.

Separate total and active parameters, then size persistent state, approximate compute and token dispatch. Compare supported parallel layouts using per-rank memory, measured communication and recovery.

Connect POST-and-poll semantics to durable item results, retry ownership and deadline capacity. Explain how cheap batch execution can still miss the delivery contract.

Find concrete failure paths in a small serving architecture. Rank correctness, admission and placement fixes before proposing additional infrastructure.

Trace event time and serving availability through a late-arriving feature. Build an executable historical join and keep online freshness separate from training correctness.