shared memory

Registers, shared memory, L1, L2, HBM, host memory: sizes, bandwidths and latencies for an H100, derived rather than recited, and the habit of asking 'which level am I hitting' before touching a kernel.

Occupancy is a means, and the end is enough bytes in flight to cover memory latency. Little's law gives the number of loads an SM needs outstanding; the register file gives the warps you can afford; the kernels that win on H100 usually run at 25% to 50% occupancy with big tiles and no spills.

Shared memory has 32 banks, each 4 bytes wide, and a warp's access is as slow as the most-loaded bank. The bank of an address, why a column walk down a 32-wide tile puts all 32 lanes in one bank, the padding by one column that spreads them across all 32, and the profiler counter that confirms the fix.

The naive kernel reads two bytes per multiply-add; a 128x128 tile with register blocking reads a few hundredths of that. Each step (shared-memory tiles, register blocking, vectorized loads, double buffering, tensor cores) with its intensity arithmetic and speedup, and where a hand kernel stops and cuBLAS begins.

A transpose moves every byte exactly once, so a device copy is the honest ceiling. Why the naive kernel pays eight times the write traffic, why staging through shared memory does nothing until you pad, and the counter that tells the two problems apart.

A row of logits is half a megabyte, so the kernel is not bandwidth-bound and the cost is in how many passes you make over it. Why a full sort does far more work than the question asks, the two shapes that fit small and large k, and the four-pass radix select with a reference that matches a sort exactly.