← AI Infrastructure System Design / 13

hardNVIDIAPerplexity

Design a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.

A reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

A reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems Design🔒 Premium

Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.

🚀 Inference & Serving🔒 Premium

The Serving Optimization WaterfallTune an LLM deployment one change at a time against one fixed workload, in an order that follows the binding constraint. A measured run went from 789 to 6,568 output tokens per second, and the two biggest wins were not the knobs people usually reach for first.

🚀 Inference & Serving🔒 Premium

Hybrid State Checkpoints and RollbackA hybrid model must restore attention history and recurrent state to the same token boundary. Learn why a cache hit can require replay, how rejected draft tokens corrupt an in-place state, and what saved checkpoints cost.

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterSeparate executor memory, allocated cache pools and logical occupancy. Map context, sequence and token-batch limits to the constraints they control, then tune a pinned deployment against per-rank memory and latency.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the registry and deploy pipeline for model weights: 141 GB artifacts, versioning, distribution to a thousand nodes, integrity.Next in this trackDesign rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?Next in this trackDesign a multi-region inference deployment: capacity per region, routing, failover, and getting the weights everywhere.

FEDITOR'S NOTE

Derive state from explicit model geometry, compare complete reload paths, and keep incomplete or incompatible cache entries out of attention.

DISCUSSION · 0

No comments yet — be the first to share your approach.