vLLM KV Cache CPU Offloading: When a Host Tier Pays, With Measurements
What vLLM's KV cache offloading does, the working-set condition under which it helps, how to enable it, and a measured RTX 5070 run where revisit time to first token fell from 77 ms to about 25 ms.
BY MAYA CASTILLO · AIINFRAINTERVIEWS EDITORIAL · UPDATED NOVEMBER 6, 2026 · 4 MIN READ
PRACTICE THIS:KV cache offloading and tiered KV storage ·Measure a CPU KV tier yourself ·LLM inference and serving questions ·KV cache size calculator
KV cache offloading lets an inference engine keep attention state that no longer fits in GPU memory. vLLM copies each completed block of the KV cache to host memory as it is produced. When a later request shares that prefix and the GPU copy has been evicted, the engine loads the blocks back over PCIe instead of recomputing the prompt. It helps when users revisit more prefixes than the GPU cache can hold, but fewer than the host tier can. In a measured run on an RTX 5070, revisits that had been recomputed at about 77 ms to first token were served in 23.5 to 26.8 ms once a 4 GiB host tier was added.
Why a GPU prefix cache can sit at zero hits
A prefix cache evicts the least recently used blocks when it fills. If requests cycle through more prefixes than fit, every revisit finds its prefix already evicted, and hits drop to nearly zero. The dashboard still shows a busy, full cache.
The size of one token's state is easy to compute: 2 (keys and values) × layers × KV heads × head width × bytes per value. For Qwen3-0.6B in BF16 that is 2 × 28 × 8 × 128 × 2 = 114,688 bytes, so a 1 GiB cache holds 9,362 tokens. Six documents of about 2,750 tokens, revisited in turn, need 16,556. The KV cache calculator does the same arithmetic for other models.
What we measured
We ran vLLM 0.31.0 with a 1 GiB GPU KV cache and compared two arms in fresh servers, twice each, in alternating order. Both arms revisited the same six synthetic documents for three rounds with one client. The only difference was a 4 GiB host tier in the second arm.
| Arm | First visit | Revisits, median TTFT | Prompt tokens loaded from host |
|---|---|---|---|
| GPU cache only | about 77 ms | 76.9 to 78.4 ms | 0 |
| With 4 GiB host tier | 77.0 to 78.5 ms | 23.5 to 26.8 ms | 16,448 of 16,556 per round |
| Two documents, fits on GPU | 74.8 ms | 13.8 to 23.8 ms | 0 (GPU hits) |
Three results matter. The GPU-only arm never hit, exactly as the arithmetic predicted. The host tier served all but each document's final partial block and the new question. And a host load was still slower than a warm GPU hit, so offloading improves on recomputation, not on a cache that already fits. The first visits cost the same in both arms, even though the offload arm wrote 1.89 GB to host memory per round; with one client the stores overlapped with other work.
These are small-model, single-client numbers. Prefill for a 0.6B model is short, so fixed per-request costs make up much of the remaining time, and the ratio here is not a general speedup. The hands-on project has the scripts, the counter reconciliation and the exact environment.
When a host tier pays
Two conditions decide it. First, the revisited working set must be larger than the GPU cache and smaller than the host tier. Below the first threshold the GPU already serves hits; above the second, the host tier thrashes in the same way. vLLM's documentation advises a host tier larger than the GPU cache for exactly this reason.
Second, loading must be cheaper than recomputing. Recomputation scales with prefill FLOPs per token, roughly two per parameter plus attention. Loading scales with KV bytes per token over the effective host-to-GPU bandwidth. Large models with grouped-query attention do a lot of compute per cached byte, so a load wins by a wide margin; small models leave less room. The KV cache offloading concept works this ratio for two model shapes, and the inference and serving questions include the full reload-versus-recompute decision with queues and NVMe.
How to turn it on
In vLLM 0.31.0 the short form is:
vllm serve Qwen/Qwen3-0.6B --kv-offloading-size 4 --kv-offloading-backend native
The size is in GiB, summed across tensor-parallel ranks. It configures the OffloadingConnector with cpu_bytes_to_use. The longer --kv-transfer-config JSON form, documented in vLLM's KV offloading usage guide, exposes the block size, LRU or ARC eviction, whether decode blocks are stored and secondary tiers such as a filesystem directory. Prometheus counters vllm:prompt_tokens_by_source_total and vllm:kv_offload_total_bytes_total show whether the tier is being used.
SGLang offers the same idea as a hierarchical cache, enabled with --enable-hierarchical-cache and sized with --hicache-ratio or --hicache-size, with write-through, selective write-through and write-back policies. Defaults and flag names move between releases in both engines, so read the help output of the version you deploy.
What to check before relying on it
Measure tokens by source per request, not a request hit rate: a request with a 16-token match counts as a hit and saves almost nothing. Confirm that computed, GPU-hit and loaded tokens add up to the prompt, and that loaded bytes divided by the KV size per token give a whole number. Then load the server with realistic concurrency. Stores from misses and loads for hits share the same host-to-GPU link, and a GPU on a shared PCIe switch or a distant NUMA node sees less bandwidth than a single-card test suggests. Finally, size the host allocation against everything else on the machine: the tier is pinned memory that the operating system cannot page out.
What interviewers ask about it
Expect the working-set question first: why a cache can be full and useless at the same time. Then the arithmetic for one token of KV state, the break-even against recomputation, and the operational traps: a shared tier needs identical block hashing across replicas, and its key must carry the model revision, tokenizer and adapter. The must-know AI infra questions include the surrounding KV cache material.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
The engine copies completed KV cache blocks from GPU memory to a larger tier, by default pinned host memory, as they are produced. When a later request shares that prefix and the blocks are no longer on the GPU, vLLM loads them back instead of recomputing them. It extends the prefix cache; it does not cache answers.
In vLLM 0.31.0, pass --kv-offloading-size with a size in GiB and --kv-offloading-backend native. That configures the OffloadingConnector with cpu_bytes_to_use. The longer --kv-transfer-config form exposes more options such as eviction policy and secondary tiers. Check vllm serve --help for your installed version, because flag names have changed between releases.
When the revisited working set already fits in the GPU cache, when it is larger than the host tier too, or when nobody revisits a prefix. A host tier no larger than the GPU cache adds almost no hits. A host load is also slower than a GPU hit, so it improves on recomputation, not on a warm GPU cache.
No. In vLLM, --cpu-offload-gb moves model weights to CPU memory to fit a larger model. KV cache offloading stores attention state for reuse. The two solve different problems and can slow a server down for different reasons.
Discussion (4)
Read the counter split before the latency chart. If loaded tokens are near zero, the tier is not doing anything and any TTFT change came from somewhere else.
And check the hash seed if two replicas share a tier. Per-process random seeds give each replica different block keys and a shared tier that never hits.
Host memory is the cheap part. The cost to watch is the PCIe link under concurrency, when stores from misses compete with loads for hits.
The model-shape point is the one people miss. Prefill FLOPs per cached byte decides how much a load can win, and a small model gives you the least room.