12Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?▼mediumSGLangAnthropicOpenAI4 replies○ sign inTwo requests share 2,000 leading tokens but ask different questions afterward. Follow the shared state through a branch, calculate the reusable blocks, and distinguish memory savings from measured latency savings.Open full answer →
12Prefix caching cut your benchmark's latency in half. Why might production see none of that?▼mediumBasetenTogether AIModal4 replies○ sign inA repeated-prompt test measures one reuse pattern. Production gains depend on token-weighted reuse, routing, residency, alignment and the prefill share of latency; a falling hit rate does not uniquely identify eviction.Open full answer →
13Design a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.▼hardNVIDIAPerplexity4 replies○ sign inA reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.Open full answer →
09Implement a cache for model weights on a serving node. What makes it different from a normal LRU?▼mediumModalBaseten4 repliesunlockedBuild a byte-budget cache with reference-counted pins and admission that preserves existing entries when it fails.Open full answer →
