kv cache

Derive the weight-only shortcut, then add KV traffic and feasible eight-GPU placement. A worked latency floor shows why batch size alone cannot establish the decode bottleneck.

Use a timeline to establish launch overhead, then trace what graph replay changes. Stable buffers, page-table updates, capture-safe operations and batch buckets determine whether the serving engine can use it safely.

Follow a logical token through the block table into a physical KV page. Then combine split attention states correctly, including empty splits, and count lookup traffic at the layer and head the kernel actually processes.

Verify a drafted prefix without biasing the target distribution, then roll back rejected KV state. A six-token prefix exposes the off-by-one: nine committed tokens can have only eight valid KV entries.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

Trace one cache position through a block table, calculate the memory recovered from oversized reservations, and explain why sharing still needs careful writes and matching.

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Draw query-head sharing, count actual stored state and compare payloads with explicit model geometry and binary units.

Separate weight storage, matrix arithmetic and attention-cache precision. Choose a supported candidate using task quality, memory and service measurements.

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

Separate checkpoint memory, expert assignments and retained MLA state. The active parameter count alone cannot predict per-rank fit or token latency.

Calculate retained cache and prefill work for a stated 128k model shape, then choose scheduling and memory policies from measured limits.

Derive a deployment candidate from the official tensor index, then account for latent-cache replication and the product workload. A worked example shows why enough aggregate HBM can still leave every GPU out of memory.

Find which startup step runs out of memory, count the allocations on that GPU, and change the setting that controls the shortage.

Derive a prompt-plus-output contract, calculate per-rank state capacity, and measure the concurrency that meets latency and overload targets.

Confirm that requests are waiting for usable state capacity. Separate allocated memory, live blocks and evictable prefixes, then compare admission, lower-precision state, prefix sharing and hardware changes with explicit assumptions.

A context-window limit is not a latency or concurrency guarantee. Budget retained state, prefill and shared service time; distinguish cache reuse from eliminating work; choose isolation only when measured interference justifies its cost.

Budget growing attention state, bounded recurrent state and retained prefix snapshots separately. Work the short-context crossover, then test whether the cache manager resumes every layer from the same prefix boundary.

Calculate Llama 3.1 70B KV cache memory at 128k context using eight KV heads. Worked BF16 and FP8 payloads, GB versus GiB, and per-GPU capacity checks.

Translate active streams into replicas using growth reservations and matched latency tests. Show peak and failure headroom separately, with no invented benchmark.

Derive the batch-295 shortcut, count the tensors it omits, and test whether the resulting kernel estimate is even inside the replica’s memory budget.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.

A reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.

Separate session state, model calls and tool execution. Calculate growing context, budget isolated workspaces and recover uncertain actions without duplicating side effects.

Size a hypothetical trillion-parameter MoE from its weight placement, attention-state layout and expert traffic. Use rack specifications as bounds, then measure the serving workload before promising throughput.

Build a reference-counted KV block allocator and test that forked sequences keep their data independent when either branch appends.

Keep live and completed candidates separate, score whole sequences, and trace a case where greedy decoding loses. Explain the cache cost of branching.

Build a small CPU reference, overwrite a bounded KV ring, and compare every step with a recomputed attention window. Keep model semantics separate from eviction.