AI Infra Interviews logo

KV cache offloading: measure what a CPU tier saves

Force a vLLM prefix cache past its capacity, add a host-memory KV tier, and measure tokens loaded, tokens recomputed and time to first token per request.

PremiumIntermediate

AI Infra Interviews editorial · Updated

How do you measure whether offloading the KV cache to CPU memory helps an inference workload?

Compute KV bytes per token for the model and compare the revisited working set with the GPU cache and the host tier. Run the same requests against an engine with the GPU prefix cache alone and one with a host tier, in fresh servers and alternating order. Record, per request, tokens computed, tokens served from GPU cache and tokens loaded from the host, and check that they sum to the prompt. A host tier helps only when the working set exceeds the GPU cache and fits the tier; a host load is still slower than a GPU hit.

Read the primary sources
What you build
Five request journals with per-request token sources and transfer bytes, plus a report whose counters reconcile.
Environment
Linux · Python 3.12 · One NVIDIA GPU · 6 GB free RAM
Plan your session
1–2 hours plus environment setup. A planning estimate, not measured runtime.
Before you begin
Complete the free streaming client project and understand time to first token. Use a single-GPU Linux host with the pinned engine environment and about 6 GB of free system memory for the host tier.
Paired GPU experiment executed. 78 measured GPU requests across five fresh servers on an RTX 5070: two sessions of each arm in alternating order plus a fits-in-GPU control. Every request reconciled token sources with its prompt and loaded bytes with 114,688 bytes per token. One client and a 0.6B model; concurrency, larger models, shared tiers and other PCIe topologies are not measured.
  1. 01Predict the cliff

    Compute KV bytes per token and compare the working set with each tier's capacity.

  2. 02Measure token sources

    Scrape per-request counters for computed, GPU-hit and host-loaded tokens in fresh servers.

  3. 03Defend the decision

    Reconcile counters, compare with a GPU-hit control and state what the run does not establish.

Follow the changing state through the project. Each boundary has a check in the walkthrough.

Continue with Premium

Premium includes this guide’s complete walkthrough and downloadable project files. The technical overview, prerequisites and verification notes above are public.

View Premium access

Start with Stream vLLM responses with Python and detect incomplete answers (Free to read).

Primary sources

Checked . Source review and execution checks are described separately above.