RunPod AI Infrastructure System Design interview questions

Eight ready slots meet a twelve-request burst. See why starting instances do not count as ready capacity, then design caches, isolation, admission and a measurable startup contract.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Trace GPU credits through reservation, allocation, metering and settlement, with crash-safe ownership and a separate fairness policy.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

Use a flexible 45-minute plan to connect requirements, a request path, capacity arithmetic, selected deep dives and a clear closing decision.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

Trace a stalled stream without unbounded telemetry, understand what sampling loses, and reconcile GPU allocation with a stated cost policy.

Follow cache hits and misses into shard calls, rerank pairs and generation work; size consistent branches and enforce access-aware reuse.

Four result rows can still hide a missing case. Build immutable evaluation identities, retry-safe commits, versioned scoring and a capacity estimate tied to measured prompt throughput.

A reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.

Derive adapter bytes from the target matrices, share compatible base weights, and keep each request's cache identity separate. A worked pool design avoids applying the mixed-adapter cost twice.

Follow GPU ownership from reservation through verified delivery and cleanup, with stale-operation fencing, recovery and auditable metering.

A 128,000-ID vocabulary breaks the 16-bit storage estimate. Follow the bytes from raw ingestion to 60 TB of token payload, then design deduplication, manifests and a restartable loader.

Bind weights, tokenizer, configuration and evaluation to one immutable release. Follow verification, prefetch, canary readiness and rollback, with bandwidth floors that do not pretend to be rollout measurements.

Remove the largest region from a concrete capacity plan. Check the surviving budget, tenant eligibility, cold prefixes and retry semantics before claiming the service can fail over.