18A new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?▼hardModalBasetenRunPod4 replies○ sign inTrace a model startup through storage, runtime setup and device loading. Transfer floors and dependencies determine whether a minute is plausible.Open full answer →
28When does offloading the KV cache to CPU memory or NVMe beat recomputing it?▼hardNVIDIAPerplexitySGLang4 replies◆ premiumA conversation's cache is leaving GPU memory. Work out whether to save it, reload it later, or recompute, including transfer queues and the chance that nobody returns.Open full answer →
09When does GPUDirect Storage actually help, and when is it just a more complicated read?▼mediumNVIDIAWEKA4 repliesunlockedCompare serialized and overlapped storage paths, then test whether direct GPU I/O improves the actual pipeline stage.Open full answer →
13Design the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?▼hardMetaMicrosoft4 replies○ sign inDesign snapshot ownership, background writes and manifest publication separately, with explicit checkpoint bytes and recovery scope.Open full answer →
13Design a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.▼hardNVIDIAPerplexity4 replies○ sign inA reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.Open full answer →
