← Inference & Serving
Advanced
KV Cache Offloading and Tiered KV Storage
KV cache offloading copies finished attention-state blocks from GPU memory to host memory, local disk or a shared store, so a revisited prefix is loaded instead of recomputed. It pays when the slower tier holds a working set the GPU cache cannot and the model's prefill work per cached byte is high.
This concept is part of Premium, which includes all concepts, course lessons and answers.
Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
AI Infrastructure System DesignDesign a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.→LLM Inference & ServingA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?→LLM Inference & ServingWhen does offloading the KV cache to CPU memory or NVMe beat recomputing it?→Open-Weights Models & Serving EnginesYou are serving an agent product with long shared prompts. vLLM or SGLang?→Open-Weights Models & Serving EnginesPrefix caching cut your benchmark's latency in half. Why might production see none of that?→LLM Inference & ServingHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?→