← AI Infrastructure System Design / 22

hardAnthropicOpenAIModal

Design the runtime for long-running agents: sessions, tool sandboxes, KV that lives for hours, and checkpoints.

Separate session state, model calls and tool execution. Calculate growing context, budget isolated workspaces and recover uncertain actions without duplicating side effects.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Separate session state, model calls and tool execution. Calculate growing context, budget isolated workspaces and recover uncertain actions without duplicating side effects.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🚀 Inference & Serving🔒 Premium

Prefix Caching and KV ReusePrefix caching reuses compatible attention state for leading tokens. Calculate shared memory and token reuse, then weigh cache identity, block ownership and routing delay before expecting a latency gain.

🔌 Networking & StorageSign in

Parallel Filesystems vs Object StorageChoose storage by access semantics, working-set reuse and failure protection. Trace a shared read/checkpoint budget, distinguish local completion from durable export, and learn when a second storage tier earns its cost.

📐 AI Systems Design🔒 Premium

Multi-Tenant Fine-Tuning ServiceA fine-tuning service turns a tenant’s dataset into an evaluated model or adapter. Its pipeline must protect data while scheduling training and serving the result efficiently.

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign serving for a video generation model: diffusion steps, batching, memory, and a latency profile unlike an LLM's.Next in this trackDesign speculative decoding into a production serving fleet: draft placement, acceptance monitoring, and the batch regime where it pays.Next in this trackDesign quota and fairness for a shared research cluster: hierarchical quotas, preemption, and the incentives that keep it honest.

FEDITOR'S NOTE

Assess durable session ownership, recoverable workspaces, measured cache benefit and safe recovery from an uncertain tool result.

DISCUSSION · 0

No comments yet — be the first to share your approach.