Describe the GPU memory hierarchy. Where can a byte live on an H100, and what does each level cost?

TL;DR: Distinguish registers, explicitly managed shared memory, hardware-managed L1/L2 caches and HBM backing. On H100, each streaming multiprocessor (SM), a group of execution units, has its own register and shared-memory resources. L2 cache and high-bandwidth memory (HBM) serve the chip. Diagnose an access pattern by counting useful bytes and memory transactions, then measuring which cache or memory boundary actually carried them.

How to approach it

Name each storage resource, its scope and who manages it. Use H100 capacities as concrete anchors without promising fixed cycle costs. Then walk a warp's load pattern through the transaction count and explain the profiler evidence needed before proposing tiling, layout changes or more bandwidth.

A strong answer

A matrix kernel requests the same number of values before and after a layout change, but its runtime improves. The arithmetic did not change. The question is whether those loads now use fewer memory transactions, hit a nearer cache, or expose more work in parallel.

The GPU Memory Hierarchy separates those possibilities:

NVIDIA's Hopper guide lists 65,536 32-bit registers per SM: 65,536 × 4 = 262,144 bytes = 256 KiB. It distinguishes the 256 KB combined L1/texture/shared pool from the 228 KB maximum shared carveout and the 227 KB per-block limit. Those are not three interchangeable capacity figures. It also lists 50 MB L2. The SXM specification gives 80 GB HBM and 3.35 TB/s peak bandwidth.

I would not quote one-cycle registers or a fixed HBM latency as a contract. An instruction dependency, a cache miss under contention and a streaming bandwidth test measure different things. Capacity, latency and bandwidth need separate definitions.

The hierarchy names possible traffic boundaries. The drawing follows a cached global load; instructions and cache policies can alter the path. It does not assign fixed cycle costs to the levels.

Now trace a load. Take a full warp of 32 lanes, each reading one four-byte value, with an aligned base address. NVIDIA's coalescing rules for compute capability 6.0 and later count the 32-byte segments needed to serve the warp.

Useful bytes in either pattern = 32 lanes × 4 bytes = 128 bytes

Adjacent words: lane i reads byte address 4i
  addresses 0, 4, ... 124 fit in four 32-byte sectors
  sector volume = 4 × 32 = 128 bytes

128-byte stride: lane i reads byte address 128i
  addresses 0, 128, ... 3968 occupy 32 separate sectors
  sector volume = 32 × 32 = 1,024 bytes

sector-volume ratio = 1,024 / 128 = 8
strided useful fraction = 128 / 1,024 = 12.5%

Sanity check: eight adjacent four-byte words fit in one 32-byte sector, so 32 adjacent lanes need four sectors. The strided case uses only one word in each requested sector. Counting an entire 128-byte cache line for every request would overstate this sector-level comparison.

These are transactions for the stated warp access, not an HBM benchmark. Cache hits, reuse across warps and other execution costs change the physical DRAM traffic and runtime. The Roofline Model needs bytes at a declared boundary, not whichever byte count gives the nicest result.

If a profiler shows poor load efficiency for the strided pattern, I would test a layout or thread-to-data mapping that makes neighboring lanes access neighboring words. I would reverse that choice if it damages the rest of the kernel enough to lose end-to-end time. Compare useful output, load sectors, cache hits, DRAM bytes and runtime on the same workload. Nsight Compute's memory chart separates traffic and behavior across memory subunits.

Moving reuse to shared memory is another option, but it adds staging and synchronization. Reducing register usage is also conditional: a smaller register allocation might allow more resident warps, or it might introduce spills. A register-only estimate at 128 registers per thread is 65,536 / 128 = 512 threads = 16 warps, only a quarter of H100's 64-warp maximum. Allocation granularity and other resources can lower actual residency further; this does not prove lower occupancy is the bottleneck.

Two naming traps matter. CUDA local memory is thread-private address space with device-memory backing and caching; it is not fast shared SRAM. And L1 cache residency is not equivalent to explicitly allocating shared memory. A working set smaller than L2 is merely a candidate for reuse: associativity, contention and access order affect whether it stays cached.

What interviewers probe next

• “All lanes read the same word. Is that worse?” For this global-load example, the requested word fits one sector. Shared-memory reads of one location also have a broadcast case; do not call every same-bank read a conflict.
• “Can I pin the working set in L2?” Persistence controls influence caching but are not a general guarantee of permanent residency. Check the supported policy and observed hits under contention.
• “Will pinned host buffers achieve peak PCIe bandwidth?” No guarantee. They avoid pageable-memory staging costs in relevant transfer paths, but size, topology, protocol overhead and overlap still matter.
• “What did Hopper add?” TMA and thread-block clusters extend data movement and cooperation. Hardware-accelerated asynchronous copies were already introduced with Ampere.

Common mistakes

• Conflating shared-memory capacity with the combined L1/shared pool.
• Reporting the strided example as a 32× HBM or runtime penalty.
• Assuming every spill misses caches or that every nominally fitting set stays in L2.
• Treating register occupancy or a hardware bandwidth peak as achieved performance.

Key takeaways

• Name scope, control and backing for each resource before quoting a cost.
• The example's adjacent load uses four sectors; the strided load uses 32, an eightfold sector-volume difference.
• The register-only bound at 128 registers/thread is sixteen warps, not a runtime prediction.
• Optimize from observed traffic and elapsed time; retain changes only when the full workload improves.