mediumOpenAI

Design the playground behind an LLM API: streaming, sessions, quotas and abuse, for a million free-tier users.

Separate durable conversation state from live generation, bound streaming buffers, and make reconnect, cancellation and free-tier budgets explicit.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Separate durable conversation state from live generation, bound streaming buffers, and make reconnect, cancellation and free-tier budgets explicit.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Tracing Inference RequestsConnect gateway, queue, model execution and streaming delivery without confusing overlapping spans or GPU activity with user-visible latency.

Advanced

🚀 Inference & Serving🔒 Premium

Voice Turns, Barge-In and Playback BuffersVoice interruption must stop unwanted generation and invalidate queued playback. Track turn identity, bound audio in milliseconds, and distinguish generated, submitted and actually played output.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Multi-Tenancy, Quotas and Fair ShareFollow three teams through GPU borrowing and reclaim. Distinguish quotas, reservations and fair sharing; see why a returning team may leave another above its nominal share, and why failures or indivisible jobs change what can be promised.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Gang Scheduling with Kueue and VolcanoGang scheduling treats a distributed job’s required workers as one admission unit. It prevents partial allocations from holding resources while waiting for the rest of the job.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the checkpoint store for a lab running several large training jobs: write bursts, retention, resharding and lineage.Next in this trackDesign observability for an inference fleet: per-request spans, GPU metrics, and cost attribution per tenant.Next in this trackDesign the control plane for a GPU cloud: tenancy, provisioning, health, billing, and the API a customer sees.

FEDITOR'S NOTE

Scored on explicit resource arithmetic, durable identity and state boundaries, failure recovery, and a design choice with its reversal condition.

DISCUSSION · 0

No comments yet — be the first to share your approach.