GPU & Accelerator Architecture: the practice test

Threads are grouped into blocks and blocks into a grid, but the streaming multiprocessor issues instructions per warp: 32 threads that execute the same instruction on different data. A block is pinned to one SM for its lifetime and split into warps; the warp scheduler swaps among resident warps to hide memory latency. That is why divergence inside a warp serializes work and why occupancy is counted in warps per SM.

Registers are private to a thread and accessed at core speed; shared memory is on-chip per SM with latency in the tens of cycles and terabytes per second of aggregate bandwidth; the L2 is a chip-wide cache of tens of megabytes with a few hundred cycles of latency; HBM is off-chip at about 3.35 TB/s on an H100 with latency in the hundreds of nanoseconds. Kernel optimization is largely the art of moving a byte as few times as possible down that hierarchy.