hardNVIDIAPerplexitySGLang

When does offloading the KV cache to CPU memory or NVMe beat recomputing it?

A conversation's cache is leaving GPU memory. Work out whether to save it, reload it later, or recompute, including transfer queues and the chance that nobody returns.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

A conversation's cache is leaving GPU memory. Work out whether to save it, reload it later, or recompute, including transfer queues and the chance that nobody returns.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

NVLink, NVSwitch and PCIeNVLink, NVSwitch and PCIe provide different GPU communication paths. Learn how their topology and bandwidth affect the placement of parallel work.

Foundational

🧩 GPU & Accelerator Architecture

GPU Memory HierarchyTrace storage scope and data reuse from registers through shared memory, caches and HBM. Worked register and traffic examples explain why a source-level load is not automatically an HBM transfer.

Foundational

⚡ Kernels & Compilers

Kernel FusionKernel fusion combines operations so intermediate results can stay on-chip. It can reduce memory traffic and launch overhead, but may increase resource pressure.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA customer sends the same prompt twice at temperature zero and gets different answers. Explain why, and what you can promise them.Next in this trackDesign an inference platform for 10,000 requests per second on a 70B model. Size it and name the SLOs.Next in this trackYour speech model runs faster than real time. Why do users still wait or hear gaps?

FEDITOR'S NOTE

Separate the decision to reload saved KV from the earlier decision to retain it. Compare complete transfer paths with recomputation, including write cost and the chance of reuse.

DISCUSSION · 0

No comments yet — be the first to share your approach.