hardNewAnthropicOpenAIModal

Design the runtime for long-running agents: sessions, tool sandboxes, KV that lives for hours, and checkpoints.

An agent thinks for a second, runs a tool for a minute, and repeats a hundred times over an hour. The session state machine and where each piece lives, the sandbox per session with its cold-start budget, the KV cache that should survive the tool call, and checkpoints that resume a step rather than an hour.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

Prefix Caching and KV ReuseMost requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.

Core

🔌 Networking & StorageSign in

Parallel Filesystems vs Object StorageA training cluster's storage has two very different jobs: stream terabytes of training data to thousands of GPUs at a steady rate, and absorb a multi-terabyte checkpoint burst every few minutes. Parallel filesystems (Lustre, GPFS, WEKA, VAST, FSx) give POSIX semantics and hundreds of GB/s of aggregate throughput; object storage (S3 and its equivalents) gives durability and cost at a fraction of the price with high first-byte latency. Almost every real cluster uses both, and the interview question is which job goes where and how big each tier has to be.

Advanced

📐 AI Systems Design🔒 Premium

Multi-Tenant Fine-Tuning ServiceA fine-tuning service takes a customer's dataset and a base model and returns a model, and the design problem is that many customers want this at once, cheaply, without seeing each other's data, on GPUs that must not sit idle between jobs. LoRA changes the shape: an adapter is a few hundred megabytes rather than a copy of the base, so many jobs can share a base in memory and many adapters can be served from one replica. This page designs the service end to end: the pipeline, the LoRA arithmetic that sets memory and cost, the isolation, the scheduler that packs jobs, and the serving path.

Core

🗂️ Scheduling & OrchestrationSign in

MIG, MPS and Time-SlicingA whole H100 is far more than a notebook, a small inference service or a CI job needs, and giving each of them a card leaves most of the fleet idle. Three mechanisms share a GPU, and they differ in what they isolate: MIG partitions the hardware into up to seven slices with their own memory and compute, MPS lets several processes share one GPU's SMs concurrently with no memory isolation, and time-slicing context-switches between processes with no isolation at all. The choice is the isolation the workload needs against the utilization the platform wants.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign serving for a video generation model: diffusion steps, batching, memory, and a latency profile unlike an LLM's.Next in this trackDesign speculative decoding into a production serving fleet: draft placement, acceptance monitoring, and the batch regime where it pays.Next in this trackDesign quota and fairness for a shared research cluster: hierarchical quotas, preemption, and the incentives that keep it honest.

FEDITOR'S NOTE

Scored on separating the three lifetimes (model call, tool execution, session), on sandbox isolation with a cold-start and cost number, on treating the agent's context as a growing prefix the KV tier serves, and on checkpointing the session at step boundaries.

DISCUSSION · 0

No comments yet — be the first to share your approach.