AI Infra Interviews logo
← Open-Weights Models & Serving Engines / 12
mediumBasetenTogether AIModal

Prefix caching cut your benchmark's latency in half. Why might production see none of that?

A repeated-prompt test measures one reuse pattern. Production gains depend on token-weighted reuse, routing, residency, alignment and the prefill share of latency; a falling hit rate does not uniquely identify eviction.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

A repeated-prompt test measures one reuse pattern. Production gains depend on token-weighted reuse, routing, residency, alignment and the prefill share of latency; a falling hit rate does not uniquely identify eviction.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational
🧮 Open Weights & Serving Engines
SGLang Server Arguments That MatterTune static memory, prefill chunks and admission against measured workload behavior. Separate offline throughput heuristics from interactive queue budgets, and calculate the part of request work that prefix caching can save.
Foundational
🧮 Open Weights & Serving Engines
Serving Benchmarks That Do Not LieDesign reproducible serving experiments with realistic arrivals, cache state and failure accounting. Interpret latency-qualified throughput and conditional per-rank bounds without turning optimistic kernel estimates into production capacity.
Advanced
🚀 Inference & Serving🔒 Premium
The Serving Optimization WaterfallTune an LLM deployment one change at a time against one fixed workload, in an order that follows the binding constraint. A measured run went from 789 to 6,568 output tokens per second, and the two biggest wins were not the knobs people usually reach for first.
Foundational
🧮 Open Weights & Serving Engines
vLLM Server Arguments That MatterSeparate executor memory, allocated cache pools and logical occupancy. Map context, sequence and token-batch limits to the constraints they control, then tune a pinned deployment against per-rank memory and latency.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackWould you split prefill and decode onto separate pools for a 753B model? Work the decision.Next in this trackA multi-node vLLM deployment starts, forms a cluster, and hangs on the first request. Debug it.Next in this trackYour product forecasts 12,800 output tokens per second at peak. Size the fleet.
FEDITOR'S NOTE

Scored on explicit cache-state experiments, request-versus-token hit denominators, reuse distance and routing, conditional latency arithmetic and evidence that separates eviction from other misses.

DISCUSSION · 0

No comments yet — be the first to share your approach.