mediumNewOpenAI

Design the playground behind an LLM API: streaming, sessions, quotas and abuse, for a million free-tier users.

The playground is the API's most public client and its most abused one: free, barely identified, streaming. The transport and its cost per connection, session state that survives a refresh with no GPU holding anything, quotas per user and per fleet, and the abuse controls that stop the free tier becoming a free API.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

💻 Coding for Infra🔒 Premium

Parsing Kernel Traces and LogsThe profiler exported a 40 GB trace; the fleet emitted a terabyte of logs overnight; the interviewer hands you a text file of kernel records and asks which kernels dominated, per GPU, per stream. The problem is a parser plus an aggregation, and it is a test of three habits: streaming instead of loading, choosing the key you aggregate on before you write a line, and handling malformed input as data rather than as an exception. This page works the reported trace-classification problem end to end, derives the memory bounds of each design, and shows the generator-based structure that scales from a screen-sized file to a fleet.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Multi-Tenancy, Quotas and Fair ShareA shared GPU pool is cheaper than ten private ones because ten teams' demand is smoother than one team's, and it only works if the sharing is enforced. Quotas say what each team is guaranteed, borrowing lets idle guarantees be used by others, fair share decides who waits when everyone wants more, and preemption reclaims borrowed capacity. This page works the arithmetic that makes pooling worth it, the layers of isolation a tenant needs, and the incentive problems (hoarding, gaming, the research-versus-product tension) that any policy has to survive.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Gang Scheduling with Kueue and VolcanoA distributed training job is 64 pods that start together or not at all: if 40 are running and 24 are Pending, the 40 hold their GPUs idle at a collective barrier waiting for ranks that may never come, and two such jobs can deadlock a whole cluster. Gang scheduling makes the job the unit of admission. Kueue and Volcano add queues, quotas, priorities and preemption on top, which is what turns a pile of GPUs into a platform several teams can share without starving each other.

Foundational

📐 AI Systems Design

Inference Platform ArchitectureAn LLM inference platform is the layer between a product's API call and a GPU running a serving engine, and every design round starts from its reference shape: a gateway that authenticates and rate-limits, a router that picks a replica with the right model and a warm cache, a per-replica scheduler that batches, engines that run prefill and decode, a KV cache tier, an autoscaler, and the observability that makes it operable. This page draws that shape, sizes each box for a concrete workload, and walks the derivation from user demand to replica count that every design answer has to contain.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the checkpoint store for a lab running several large training jobs: write bursts, retention, resharding and lineage.Next in this trackDesign observability for an inference fleet: per-request spans, GPU metrics, and cost attribution per tenant.Next in this trackDesign the control plane for a GPU cloud: tenancy, provisioning, health, billing, and the API a customer sees.

FEDITOR'S NOTE

Scored on choosing the streaming transport with the connection arithmetic, on session state kept off the GPU and out of the engine, on quotas per user and per fleet, and on abuse controls that treat the playground as an untrusted API client.

DISCUSSION · 0

No comments yet — be the first to share your approach.