03Why are GPUs so much faster than CPUs for deep learning? Be specific about what the silicon is doing differently.▼easyNVIDIAGoogle4 repliesunlockedCompare useful parallel work, data movement and reuse. A synthetic CPU/GPU crossover shows why a faster kernel can still lose the end-to-end request.Open full answer →
08How much faster is an H200 than an H100, really? Which workloads see the gain and which do not?▼mediumNVIDIACoreWeaveLambda4 repliesunlockedSeparate H200 bandwidth and capacity gains from measured speedup, then test realistic memory headroom and cost per useful completed workload.Open full answer →
12Explain what HBM is and why memory bandwidth, not compute, is the wall for LLM inference.▼mediumNVIDIAAMDMicron4 replies○ sign inDerive HBM bandwidth from interface width and pin rate, then connect it to a model that actually fits. Separate raw interface capacity from measured decode time.Open full answer →
29If you could change one thing about GPU architecture for LLMs, what would it be, and what argues against it?▼expertNVIDIAAnthropicOpenAI4 replies◆ premiumA proposal for more memory bandwidth needs a workload that benefits, a budget for what it costs, and a reason to abandon it. Work through a serving-specific design without assuming all LLM work has the same bottleneck.Open full answer →
26nvidia-smi shows 30 percent utilization on our serving fleet. Is that a problem, and what would you look at instead?▼mediumBasetenModal4 replies◆ premiumThirty percent GPU utilization means kernels ran during about thirty percent of the sampled time. Find out what limits those kernels, what happens between them, and whether requests are meeting their latency targets.Open full answer →
