kv cache

At batch 1 a decode step reads every weight byte to do two FLOPs with it. The intensity is the batch size, the ridge is about 295, and the KV cache puts a ceiling on how far batching can push you. The full chain for a 70B model on H100.

At batch 1 a decode step is a few milliseconds of GPU work behind several hundred kernel launches, and the CPU cannot issue them fast enough. The arithmetic showing the GPU starve, what replay changes, and the four constraints capture imposes on memory.

The KV for one sequence is scattered across fixed-size pages, so the kernel reads a table of physical block numbers before it can read any keys. What the indirection costs, what the table costs in memory, why decode must split the KV dimension across thread blocks, and how the partial softmax states combine.

The target runs one forward pass over all k drafted tokens, compares its distribution against the draft's at each position, and keeps the longest prefix that survives a coin flip. The acceptance rule that makes the output exactly a target sample, the resampling at the first rejection, and the one-integer rollback.

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.

Weight-only int4, fp8 weights and activations, or fp8 for the cache alone: each fixes a different bottleneck. Match the method to the resource you are short of, then run the evals that catch what perplexity misses.

A flat median with a broken tail means one in a hundred requests is hitting something the others do not. There are five usual suspects, and the order you rule them out in is the answer.

671 billion parameters that touch 37 billion per token: the memory bill is for the big number and the bandwidth bill is for the small one, until batch and routing collide. The cache is tiny, and that changes the whole fit.

At 128k tokens one request's cache is 43 gigabytes and its prefill is measured in seconds. Capacity breaks first, then TTFT, then the scheduler. Each has a fix, and the numbers say which fix you need at which length.

The answer comes from the model card and config.json in five minutes, before downloading anything. The weight footprint, the KV per token that is far smaller than the parameter count suggests, and the parallel degree that has to divide the head count.

Failing at startup and failing under load are different problems with different fixes, and the engine tells you which in its own log. Where the memory actually goes, the four flags that move it, and the arithmetic that says whether it can fit at all.

These two flags decide how much memory the engine reserves and how many users share each step, and both are usually left at values that come from the model rather than from the product. The derivation, the SLO that bounds the second one, and the measurement that confirms both.

A full cache pool with idle streaming multiprocessors is a specific signature with a specific set of remedies. Six of them, ranked by what they cost, and the one that is free because the flag was set from the model rather than the product.

Four things break in order and only one of them is the attention math everyone expects. What a single million-token sequence costs in memory, why the first token takes minutes, and what the model's own architecture already did about the parts that would otherwise be impossible.

Most layers stop having a growing cache, which changes capacity planning by a large factor and demands a cache manager most engines did not have. What the memory model becomes, what the engine must implement, and the two features that were disabled by default while it settled.

Four numbers from the config file, one formula, and a per-sequence result that is a third of the model's own weights. Plus the mistake that makes the answer eight times too big.

Two constraints per replica, memory and throughput, each derived from the model and the traffic; whichever binds sets the users per replica, and the replica count follows with headroom. The full chain from 1,000 users to a number of nodes.

For bf16 weights the decode intensity equals the batch, so the ridge is at batch 295 on an H100. Then the KV cache enters and the answer changes shape: the projections go compute-bound at 300 while the attention reads never do, and at 8k context the step as a whole stays memory-bound at any batch.

Seven boxes between an API call and a GPU, each with one job and one way to fail. The walkthrough a screen expects in the first ten minutes, with the sizing chain from 2,000 concurrent users to a replica count so the drawing has numbers on it.

From an admission queue to a streamed token, the batching system that decides how many tokens per second a replica earns and what its p95 TPOT is. The decode arithmetic that sets the batch, the KV budget that caps it, and the streaming path that must never stall the engine.

A 70B model's KV cache is 320 KB per token, and a 30k-token conversation is 10 GB that GPU memory cannot keep between turns. The three tiers with their bandwidths, the break-even where reloading a prefix beats recomputing it, the eviction and lookup design, and the traffic shape where the tier is worth its complexity.

An agent thinks for a second, runs a tool for a minute, and repeats a hundred times over an hour. The session state machine and where each piece lives, the sandbox per session with its cold-start budget, the KV cache that should survive the tool call, and checkpoints that resume a step rather than an hour.

A trillion parameters with 32 billion active per token: the weights need a rack, the compute needs a fraction of it, and the all-to-all between them decides everything. The memory plan across 72 GPUs, the decode step from bandwidth and two all-to-alls, the compressed attention cache, and what imbalance costs.

A free list, a per-sequence block table, and a reference count are the whole structure. What forking costs, why the first write after a fork must copy, and the measured waste that this design accepts in exchange for eliminating the far larger waste of pre-allocation.