tensor cores

Compare useful parallel work, data movement and reuse. A synthetic CPU/GPU crossover shows why a faster kernel can still lose the end-to-end request.

Follow operand tiles into a matrix-multiply-accumulate instruction. Explain why odd dimensions and low tensor activity do not prove a fallback.

Derive how reusing weights changes arithmetic intensity, then separate the roofline ceiling from tile-wave effects and measured GEMM throughput.

Separate valid 2:4 structure, packed representation and sparse dispatch from model quality and the fraction of request time that can improve.

Follow an accumulator across K tiles, count reuse at each memory level, and decide whether a tile improves measured GEMM performance.

Compare the mechanisms in FlashAttention versions 1, 2 and 3: IO savings, work partitioning, asynchronous dependencies and FP8 accuracy.

Separate FP8 input quantization from accumulation error. A small representable-value example shows when a shared scale loses small values, while the DeepSeek-V3 report supplies one carefully scoped training recipe.