12Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?▼mediumNewSGLangAnthropicOpenAI4 replies○ sign inThe same 2,000 tokens are prefilled a thousand times an hour. Caching their KV by content turns that into one prefill and a table lookup, and a radix tree is what makes multi-turn and branching agents share it too.Open full answer →
12Prefix caching cut your benchmark's latency in half. Why might production see none of that?▼mediumNewBasetenTogether AIModal4 replies○ sign inA benchmark that replays one prompt measures the cache and reports a number production will never see. What the hit rate actually depends on, the eviction behaviour that erodes it under load, and the measurement that predicts the real gain.Open full answer →
13Design a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.▼hardNewNVIDIAPerplexity4 replies○ sign inA 70B model's KV cache is 320 KB per token, and a 30k-token conversation is 10 GB that GPU memory cannot keep between turns. The three tiers with their bandwidths, the break-even where reloading a prefix beats recomputing it, the eviction and lookup design, and the traffic shape where the tier is worth its complexity.Open full answer →
09Implement a cache for model weights on a serving node. What makes it different from a normal LRU?▼mediumNewModalBaseten4 repliesunlockedEntries differ in size by a factor of fifty, so a count-based cache is meaningless, and evicting the model currently serving requests is a correctness bug rather than a performance one. Size accounting, pinning, and the two failures that must raise rather than silently evict.Open full answer →
