hardNewNVIDIAPerplexity

Design a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.

A 70B model's KV cache is 320 KB per token, and a 30k-token conversation is 10 GB that GPU memory cannot keep between turns. The three tiers with their bandwidths, the break-even where reloading a prefix beats recomputing it, the eviction and lookup design, and the traffic shape where the tier is worth its complexity.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Foundational

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterA vLLM deployment is mostly decided by a dozen flags, and the ones that matter fall into four groups: how the model is split across GPUs, how memory is divided between weights and cache, how requests are batched, and which specialized backends the model needs. Getting the first two wrong produces an engine that will not start or that runs out of memory under load. Getting the third wrong produces an engine that starts, serves, and misses its latency target by a wide margin.

Foundational

🧮 Open Weights & Serving Engines

SGLang Server Arguments That MatterSGLang's tuning model is different from vLLM's in one way that matters: it exposes the scheduler's aggressiveness and the static memory fraction as direct knobs, and its own documentation gives target values for the runtime signals those knobs move. That makes tuning it a measurement loop rather than guesswork. Aim for a queue of a hundred to a couple of thousand requests, token usage above 0.9, and five to eight gigabytes of free GPU memory after startup, then adjust the flags that move each one.

Advanced

🚀 Inference & Serving🔒 Premium

Prefix Caching and KV ReuseMost requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the registry and deploy pipeline for model weights: 141 GB artifacts, versioning, distribution to a thousand nodes, integrity.Next in this trackDesign rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?Next in this trackDesign a multi-region inference deployment: capacity per region, routing, failover, and getting the weights everywhere.

FEDITOR'S NOTE

Scored on the per-token KV number, on the break-even between reloading KV from a tier and recomputing prefill, on the block-hash lookup and eviction design, and on naming the traffic (multi-turn, shared long documents) where it pays.

DISCUSSION · 0

No comments yet — be the first to share your approach.