system design
AI infra interview questions tagged system design, across every topic.
3 questions · 1 unlocked for you
Concepts behind "system design"
The curriculum that explains the ideas these questions test.
Foundational
Inference Platform ArchitectureAn LLM inference platform is the layer between a product's API call and a GPU running a serving engine, and every design round starts from its reference shape: a gateway that authenticates and rate-limits, a router that picks a replica with the right model and a warm cache, a per-replica scheduler that batches, engines that run prefill and decode, a KV cache tier, an autoscaler, and the observability that makes it operable. This page draws that shape, sizes each box for a concrete workload, and walks the derivation from user demand to replica count that every design answer has to contain.📐 AI Systems Design
Foundational
The AI Infra Design Round PlaybookThe AI infrastructure design round is 45 to 60 minutes with one prompt (design a serving platform, a training scheduler, a 10k-GPU cluster, a fine-tuning service) and one interviewer whose job is to find the edge of what you know. The candidates who pass do the same things in the same order: pin the requirements and the numbers in the first five minutes, draw the reference shape, size it with a stated chain of arithmetic, pick two deep dives, and name the failure modes before being asked. This page gives that structure with a minute-by-minute plan, the numbers to bring in your head, and the mistakes that end the round early.📐 AI Systems Design
Core
GPU Job Scheduler DesignDesign a scheduler for a shared GPU cluster is the most common design prompt in AI infrastructure interviews, because it touches everything: queues and priorities, gang placement, topology, fairness across teams, preemption and the checkpoints that make it survivable, and the failure handling that keeps a 512-GPU job alive. This page builds the design in layers, states the data model and the scheduling loop, derives the numbers (how long a job waits, how much preemption costs, how much fragmentation wastes), and lists the trade-offs the interviewer will push on.📐 AI Systems DesignSign in
Advanced
Training Cluster Design at 10k GPUsDesign a cluster for training frontier models is the prompt that tests whether a candidate can hold hardware, network, storage, scheduling and reliability in one head at once. The answer is a bill of materials with a reason for every line: how many GPUs and why, how they are grouped into pods, how the fabric connects the pods and what it costs a collective to cross one, how much storage bandwidth the checkpoints and the data loader need, how power and cooling bound the whole thing, and how the failure statistics set the spare pool and the checkpoint cadence. This page derives each line for a 10,240-GPU cluster.📐 AI Systems Design🔒 Premium
Advanced
Multi-Tenant Fine-Tuning ServiceA fine-tuning service takes a customer's dataset and a base model and returns a model, and the design problem is that many customers want this at once, cheaply, without seeing each other's data, on GPUs that must not sit idle between jobs. LoRA changes the shape: an adapter is a few hundred megabytes rather than a copy of the base, so many jobs can share a base in memory and many adapters can be served from one replica. This page designs the service end to end: the pipeline, the LoRA arithmetic that sets memory and cost, the isolation, the scheduler that packs jobs, and the serving path.📐 AI Systems Design🔒 Premium