GPU Memory Hierarchy
Trace storage scope and data reuse from registers through shared memory, caches and HBM. Worked register and traffic examples explain why a source-level load is not automatically an HBM transfer.
TL;DR: Registers hold a thread's working values; shared memory lets threads explicitly cooperate; L1 and L2 cache memory accesses; HBM holds large device allocations. Ask which resource stores the data, who can reuse it, and which boundary the traffic crosses. A source-level load is not necessarily an HBM read, and moving data on-chip is useful only if its costs are repaid.
Follow a value through the work
Suppose a kernel repeatedly uses the same matrix tile. Loading each operand through a global-memory pointer is easy to write, but it leaves reuse to the cache and the access pattern. Staging a tile in shared memory can let a block of threads reuse it explicitly. Each thread then keeps operands and partial results in registers.
An SM, or streaming multiprocessor, runs groups of threads. A warp is a group of 32 threads executing instructions together on NVIDIA GPUs. Storage near the SM is small and contested by the threads resident there; device memory is much larger. The GPU Execution Model explains how those threads are scheduled.
Do not picture the hierarchy as five compulsory stops for every byte. An ordinary global load can hit a cache, a copy engine can move data directly into shared memory, and registers are not a hardware-managed cache of everything below them.
Separate capacity, scope and control
For , NVIDIA's Hopper tuning guide specifies 64K 32-bit registers per SM, a combined L1/texture/shared pool of 256 KB, up to 228 KB shared per SM and 227 KB addressable by one block. The shared-memory carveout is configurable; shared memory and L1 have different programming semantics even though they use a common physical pool. Large shared allocations also require the appropriate opt-in.
| Storage | Who uses or manages it? | What can go wrong? |
|---|---|---|
| Registers | Compiler allocates a thread's working state | High usage limits residency or causes spills |
| Shared memory | Programmer coordinates threads, normally within a block | Synchronization, bank conflicts and capacity costs |
| L1 cache | Hardware caches selected accesses near an SM | Misses, contention and access-policy differences |
| L2 cache | Hardware cache serving the chip | Data can be evicted despite fitting nominal capacity |
| HBM | Device allocation backing on this discrete GPU | Capacity or data-transfer bandwidth can limit work |
The H100 L2 is specified as 50 MB. The H100 SXM product specification lists 80 GB HBM and 3.35 TB/s peak bandwidth. Those capacity and bandwidth figures do not give a universal access latency. Dependency chains, cache state, contention and clock speed affect observed time. Do not turn approximate cycle counts into a fixed speedup table.
Registers are private, but their budget is shared
A thread cannot borrow another thread's register values by addressing them like an array in shared memory. Yet the SM has one finite register budget for all resident work:
65,536 registers × 4 bytes = 262,144 bytes = 256 KiB per SM
128 registers/thread → at most 65,536 / 128 = 512 threads
512 / 32 threads per warp = 16 warps
That is a register-only upper bound, not a promised occupancy. Block size, allocation granularity, shared memory and other limits can reduce it. H100 supports at most 64 resident warps per SM, so sixteen is 25% of that warp limit. More occupancy is not automatically faster: additional registers can avoid spills or expose useful independent work.
CUDA local memory is a misleading name for newcomers. It is private in address scope, not an on-chip scratchpad. A spill can generate local-memory loads and stores backed by device memory and caches. Inspect compiler resource reports and measured traffic before blaming every local access for an HBM round trip.
Reuse only helps when it pays for staging
For a hypothetical 16 KiB tile used eight times by one block, eight complete global read passes request 128 KiB at source level. Staging it once requests 16 KiB globally, then performs the repeated reads from shared memory. That is an eightfold reduction in requested global bytes under this simple model. It is not proof of an eightfold reduction in HBM traffic or time: the original reads may hit L1/L2, and staging adds copies, synchronization and shared-memory demand.
| Observation | Useful next check |
|---|---|
| Many redundant global reads | Test tiling; compare actual DRAM and cache traffic |
| High register demand or spills | Inspect resource use and runtime before reducing register limits |
| Poor shared-memory throughput | Check synchronization and bank conflicts |
| Working set seems to fit L2 | Compare warm and cold behavior; inspect hits and evictions |
For shared-memory reads, different addresses in the same bank can conflict. Multiple lanes reading the same location can use broadcast instead. The CUDA best-practices guide explains these access rules and global Memory Coalescing. Address pattern matters as much as where storage resides.
What Hopper adds, and what it does not promise
Ampere introduced hardware-accelerated asynchronous global-to-shared copies. Hopper adds the Tensor Memory Accelerator, which can move multidimensional tiles with less address and copy work by the threads. Completion still needs synchronization before consumers read the data. Hopper also supports distributed shared memory within a thread-block cluster, extending the ordinary block-local cooperation scope. These mechanisms are described in the Hopper guide above; neither makes data movement free.
Host and peer memory add interconnects to the path. Pinned host buffers can improve transfers and enable appropriate asynchronous copies, but they do not guarantee the advertised PCIe rate. Check transfer sizes, topology and overlap. Unified addressing also does not mean every pointer refers to equally fast local storage.
Check your understanding
The tiled version reduces global load requests eightfold, but measured HBM reads barely change. Is the measurement necessarily wrong? No. The original version may already have served most reuse from cache. Compare cache traffic, synchronization and runtime before keeping the extra shared-memory machinery.
Key takeaways
- Separate storage scope from physical backing: local memory is thread-private but can use device memory and caches.
- 128 registers per thread gives a sixteen-warp register-only bound on the stated H100 SM.
- A source-level read count is not a measured HBM byte count. Name the boundary when claiming savings.
- Tiling, fusion and asynchronous copies are experiments with costs; verify the critical path rather than memorize latency ratios.