TL;DR: Both engines support prefix reuse, so a long shared prompt is a reason to test caching, not enough to choose an engine. Verify the exact checkpoint, hardware, quantization and API features first. Then measure whether requests reach reusable cache state under real routing and memory pressure, and compare useful completions within the service-level objective (SLO). A 2,000-token hit in a 2,200-token prompt saves 90.9% of newly computed prompt positions; it does not establish a 90.9% latency saving or a GPU-count reduction.
How to approach it
Define the agent contract: tool-call correctness, streaming, context and output limits, latency and acceptable failure rate. Check both implementations against that contract. Estimate the reusable prompt work, then design a controlled comparison that preserves arrivals, conversation branches and cache pressure. End with a concrete selection rule and the evidence that would reverse it.
A strong answer
An agent product sends the same 2,000-token system prompt and tool schema, followed by 200 new tokens per turn. It expects 40 requests/s. That sounds highly reusable, but timestamps before the stable text, tool-schema changes, model revisions and routing can all reduce actual hits.
Measure tokenized matching prefixes after the chat template is applied. Record the fraction of requests with a hit and the number of reused positions separately. A one-token hit and a 2,000-token hit should not count as equivalent savings. Preserve tenant isolation and model, adapter and multimodal identity in the cache key. Prefix Caching and KV Reuse explains why matching words alone are insufficient.
Bound the benefit without inventing capacity
In this synthetic example, 80% of requests get the full 2,000-position hit and 20% get none. Ignore block rounding for the initial estimate:
Expected reused positions/request = 0.8 × 2,000 = 1,600
Expected new prompt positions = 2,200 - 1,600 = 600
Reduction in new prompt positions = 1,600 / 2,200 = 72.7%
At 40 requests/s: 64,000 prompt positions/s avoid new prefill
sanity: 600 new + 1,600 reused = the original 2,200 positions.
At a perfect hit rate, the reduction is 90.9%. With a hypothetical 32-token block boundary, only 1,984 positions of a 2,000-token prefix are full blocks; a backend that reuses only those blocks leaves 216 new positions. Use the actual engine's reuse boundary and state requirements.
The linear-layer FLOP proxy 2 × active parameters × new positions estimates one part of saved computation. Suffix attention still accesses prior state, and lookup, transfer and execution efficiency have costs. Do not divide that proxy by an advertised GPU peak and call the result GPUs saved. Replay the workload on candidate replica layouts to estimate actual deployable capacity.
Compare behavior, not the data-structure name
The SGLang paper describes RadixAttention's radix-tree reuse. vLLM's prefix-cache design uses chained block hashes incorporating the preceding prefix and other identity information. Both can reuse common heads before conversations branch. The tree alone does not prove better service latency.
| Decision dimension | Evidence needed | What could reverse the preference |
|---|---|---|
| Exact model support | Successful loading, quality and API tests with pinned revisions | A required attention, quantization or parser path fails |
| Cache value | Reused positions, TTFT and recovery under memory pressure | High hit count but short hits, slow restoration or frequent recomputation |
| Routing | Hit location, per-replica queue and transfer cost | Affinity saves prefill but creates a larger queue delay |
| Operations | Rollout, cancellation, metrics and recovery tests | Small throughput gain creates unacceptable operational cost |
| Service performance | SLO-qualified completions at the offered workload | Better peak token/s but worse latency, rejection or tool success |
SGLang Server Arguments That Matter and vLLM Server Arguments That Matter explain tuning controls. Both require measurement. SGLang's offline queue heuristics are not an intrinsic advantage for an interactive agent.
A popular prefix is not guaranteed to survive. Restart, cache pressure, a new replica or changed identity can make it cold. A hit-rate decline that correlates with load suggests several hypotheses: eviction, changed traffic, routing imbalance or a different effective prefix. Inspect reuse distance and placement before enlarging the pool. More cache does not fix requests sent to a replica that lacks the state.
Run two complementary comparisons
First isolate implementation differences with the same hardware allocation, checkpoint, precision, generation settings and arrival trace. Preserve turn ordering and realistic delays between turns. Include cold start, warm steady state, conversation branching and a working set larger than cache. Use sanitized fixtures that preserve token lengths and reuse structure when production prompts cannot be used safely.
Then allow each engine a documented tuning budget to find its best valid configuration on the same resource budget. Equivalent behavior matters more than identical flag values. Record resolved arguments and cache warmup, not just the command. Compare repeated runs, error counts, TTFT and token-gap tails, useful throughput, and tool/structured-output correctness. Include rejected and timed-out arrivals in the report.
Suppose a synthetic result gives engine A 40 completed requests/s with 95% of completions meeting the full contract, and B 38 with 99%. Their qualified completion rates are 38 and 37.62 requests/s respectively. A is only about 1% ahead on that metric, not the 5.3% suggested by raw completions. If both faced 42 arrivals/s, also report their qualified fractions of offered traffic: 90.5% and 89.6%. Repeated-run uncertainty and any stricter acceptance floor can decide whether either is acceptable. These labels do not stand for measured vLLM or SGLang results.
I would keep the engine that passes the required model/API tests and meets the service target at the best supported operating cost. If both qualify with a small uncertain difference, operational familiarity can decide. I would reverse that choice after a reproducible improvement or a required capability changes. If neither qualifies, "one supports the architecture" is not enough to deploy; evaluate another checkpoint or change the service design. No engine benchmark was run for this answer.
What interviewers probe next
- "Does a new turn reuse the whole history?" Only if the token prefix still matches, the state is retained and reachable, and the engine supports restoring it. Truncation, tool formatting or rerouting can break that chain.
- "Can caching change correctness?" Correct reuse should preserve the computation within the numerical behavior of the implementation. Test invalidation, adapter identity and hybrid-state restoration; do not assume implementation bugs are impossible.
- "Is repeated-prompt testing invalid?" It is useful as a labeled warm-cache boundary case. It is misleading when presented as representative production capacity.
- "Why compare arrivals rather than only concurrency?" Fixed concurrency reduces arrivals as responses slow, potentially hiding overload. Use it diagnostically, then test independent offered load.
Common mistakes
- Equating a prefix-sharing opportunity with an actual cache hit.
- Calling fewer new prompt positions the same percentage of total work saved.
- Attributing one engine's advantage to a feature both have.
- Ignoring routing and cold replicas in a single-process benchmark.
Key takeaways
- At 80% full hits, this 2,200-position workload averages 600 new positions, a 72.7% reduction.
- A 32-token full-block boundary would reuse 1,984 of the 2,000 matching positions.
- The synthetic A/B example narrows from 5.3% raw-throughput advantage to about 1% qualified-throughput advantage.
- Select on supported, correct service behavior under reproducible traffic and memory pressure.