18cuBLAS already gives you a fast GEMM. What does CUTLASS give you that it does not, and when is that worth it?▼hardNewNVIDIAFireworks4 replies○ sign inA library call gives you a tuned matmul and nothing around it. Four things a template library gives instead: an epilogue fused into the output stage, one launch for a list of ragged problems, a matmul whose inputs are different types, and a tile schedule that fills the SMs on awkward shapes.Open full answer →
24A mixture-of-experts layer is 256 small matmuls. Why is a loop over cuBLAS calls slow, and what does a grouped GEMM do instead?▼hardNewFireworksTogether AIDeepSeek4 replies◆ premiumRouting hands each expert a different number of tokens, so the layer is a list of ragged matmuls rather than one. Three separate costs a loop pays, the measured share of compute that skewed routing wastes on tile padding, and what a single grouped launch fixes and what it cannot.Open full answer →
