mediumBasetenOpenAI

Design observability for an inference fleet: per-request spans, GPU metrics, and cost attribution per tenant.

Trace a stalled stream without unbounded telemetry, understand what sampling loses, and reconcile GPU allocation with a stated cost policy.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Trace a stalled stream without unbounded telemetry, understand what sampling loses, and reconcile GPU allocation with a stated cost policy.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🩺 Fleet Reliability & Observability🔒 Premium

Tracing Inference RequestsConnect gateway, queue, model execution and streaming delivery without confusing overlapping spans or GPU activity with user-visible latency.

Advanced

📐 AI Systems Design🔒 Premium

Multi-Tenant Fine-Tuning ServiceA fine-tuning service turns a tenant’s dataset into an evaluated model or adapter. Its pipeline must protect data while scheduling training and serving the result efficiently.

Advanced

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.

Advanced

⚡ Kernels & Compilers🔒 Premium

Profiling with NsightNsight Systems shows where an application spends time; Nsight Compute investigates individual kernels. Use the timeline to choose what deserves detailed profiling.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the control plane for a GPU cloud: tenancy, provisioning, health, billing, and the API a customer sees.Next in this trackDesign an LLM service for a 200 ms time-to-first-token SLO at p99. Decompose the budget and say what you would give up.Next in this trackDesign the runtime for long-running agents: sessions, tool sandboxes, KV that lives for hours, and checkpoints.

FEDITOR'S NOTE

Scored on explicit resource arithmetic, durable identity and state boundaries, failure recovery, and a design choice with its reversal condition.

DISCUSSION · 0

No comments yet — be the first to share your approach.