OpenAI AI Infrastructure System Design interview questions

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Trace GPU credits through reservation, allocation, metering and settlement, with crash-safe ownership and a separate fairness policy.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

Four result rows can still hide a missing case. Build immutable evaluation identities, retry-safe commits, versioned scoring and a capacity estimate tied to measured prompt throughput.

Divide one token budget across gateways instead of copying it. Follow output reservation and settlement, then handle grants, retries and outages without inventing extra burst capacity.

Remove the largest region from a concrete capacity plan. Check the surviving budget, tenant eligibility, cold prefixes and retry semantics before claiming the service can fail over.

Separate durable conversation state from live generation, bound streaming buffers, and make reconnect, cancellation and free-tier budgets explicit.

Trace a stalled stream without unbounded telemetry, understand what sampling loses, and reconcile GPU allocation with a stated cost policy.

Separate session state, model calls and tool execution. Calculate growing context, budget isolated workspaces and recover uncertain actions without duplicating side effects.

Follow a video job from prompt to downloadable artifact. Work the latent-shape arithmetic, compare batching with actual delivery times, and design bounded queues, cancellation and safe retries.

A team returns to claim its GPUs, but another team is borrowing them in one indivisible job. Work through the quota tree, whole-job reclaim, recovery costs and release-to-ready budget before promising a start time.

Trace usage from execution to one settled charge. Distinguish requests, attempts and events, then handle cancellation, late records, replay and reconciliation.

Size a hypothetical trillion-parameter MoE from its weight placement, attention-state layout and expert traffic. Use rack specifications as bounds, then measure the serving workload before promising throughput.

Separate total and active parameters, then size persistent state, approximate compute and token dispatch. Compare supported parallel layouts using per-rank memory, measured communication and recovery.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

Use a flexible 45-minute plan to connect requirements, a request path, capacity arithmetic, selected deep dives and a clear closing decision.

Follow cache hits and misses into shard calls, rerank pairs and generation work; size consistent branches and enforce access-aware reuse.

Eight ready slots meet a twelve-request burst. See why starting instances do not count as ready capacity, then design caches, isolation, admission and a measurable startup contract.

A reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.