Run untrusted user code at 50,000 concurrent sessions, some on GPUs. Pick the isolation boundary and defend the density you lose.

Containers share a kernel with the code they run, the wrong boundary for code you did not write. gVisor, Firecracker microVMs and full VMs each buy a stronger one at a cost in memory, start time and GPU access. The overhead arithmetic that turns 50,000 sessions into a host count, and the tiered design.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

📐 AI Systems Design🔒 Premium

Serverless GPU PlatformsA serverless GPU platform lets a customer deploy a function or a model and pay only while it runs, so the platform has to start a GPU workload in seconds, pack many customers onto shared hardware without letting them see each other, and keep enough capacity warm that a burst does not wait for a cold start. Each is a design problem with numbers: the cold-start chain and the snapshot that shortens it, bin-packing memory-sized workloads onto fixed-size GPUs, the isolation boundary and its cost, and the economics of idle capacity against cold starts. This page designs the platform and derives the trade-offs.

Core

🗂️ Scheduling & OrchestrationSign in

MIG, MPS and Time-SlicingA whole H100 is far more than a notebook, a small inference service or a CI job needs, and giving each of them a card leaves most of the fleet idle. Three mechanisms share a GPU, and they differ in what they isolate: MIG partitions the hardware into up to seven slices with their own memory and compute, MPS lets several processes share one GPU's SMs concurrently with no memory isolation, and time-slicing context-switches between processes with no isolation at all. The choice is the isolation the workload needs against the utilization the platform wants.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Multi-Tenancy, Quotas and Fair ShareA shared GPU pool is cheaper than ten private ones because ten teams' demand is smoother than one team's, and it only works if the sharing is enforced. Quotas say what each team is guaranteed, borrowing lets idle guarantees be used by others, fair share decides who waits when everyone wants more, and preemption reclaims borrowed capacity. This page works the arithmetic that makes pooling worth it, the layers of isolation a tenant needs, and the incentive problems (hoarding, gaming, the research-versus-product tension) that any policy has to survive.

Advanced

📐 AI Systems Design🔒 Premium

Multi-Tenant Fine-Tuning ServiceA fine-tuning service takes a customer's dataset and a base model and returns a model, and the design problem is that many customers want this at once, cheaply, without seeing each other's data, on GPUs that must not sit idle between jobs. LoRA changes the shape: an adapter is a few hundred megabytes rather than a copy of the base, so many jobs can share a base in memory and many adapters can be served from one replica. This page designs the service end to end: the pipeline, the LoRA arithmetic that sets memory and cost, the isolation, the scheduler that packs jobs, and the serving path.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour fine-tunes run on spot GPUs preempted about once every four hours. How often should they checkpoint, and when does spot stop paying?Next in this trackOne fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.Next in this trackDesign a job queue for 100k GPU jobs with preemption: what state, what ordering, and what happens when a quota owner returns?

FEDITOR'S NOTE

Scored on naming the kernel as the shared attack surface, on giving a per-session overhead number for each boundary and multiplying it out, and on treating GPU sessions as a separate tier with its own boundary.

DISCUSSION · 0

No comments yet — be the first to share your approach.