Follow operands from memory to execution units

TL;DR: A GPU needs both places to store data and units that perform calculations on it. Tensor cores do matrix arithmetic; registers, shared memory and caches hold the values that arithmetic needs. Following a value from storage to execution helps explain why available compute units can still be waiting for work.

Where you are. The previous lesson compared arithmetic with the cost of moving weights. Now look inside the GPU to see where those values are stored and how they reach the units that use them.

Where the data lives and where the work runs

A calculation needs its input values, also called operands, before it can run. A tensor core takes matrix operands and computes a result. Registers, shared memory and caches keep data nearby so execution units can use it. If those units are waiting for inputs, adding more arithmetic capacity will not necessarily make the kernel faster. A kernel is a function launched to run on the GPU.

“Why is occupancy low?” starts on the execution side, but register and shared-memory allocations may explain the limit. “Why does this kernel read four times the required bytes?” starts with memory transactions. The columns organize the investigation; dependencies cross between them.

The execution side, in the detail that matters

GPU work is divided into threads. On the architecture described here, 32 threads form a warp, with each thread occupying a lane. When threads in a warp take different branches, the GPU executes the paths with different lanes active. This is called divergence.

Suppose 16 lanes take one arithmetic path and the other 16 take an equally expensive path. About half the lanes are active while each path runs, so lane utilization in that region is approximately 50%. That does not mean the whole kernel takes twice as long: the rest of its work, memory stalls and the compiler's treatment of branches also matter. Threads that exchange data still need explicit synchronization wherever the programming model requires it; belonging to the same warp is not enough to assume they stay synchronized.

Warps run on streaming multiprocessors, or SMs. Each SM has registers, shared memory and execution units. It keeps several warps resident, meaning their execution state is available on that SM. When one warp waits for data, another ready warp can issue an instruction. This hides some of the waiting time, provided enough independent work is available.

Tensor cores sit inside the SM alongside the general arithmetic units. General units support scalar and vector operations; matrix instructions use specialized hardware to multiply and accumulate tiles. A tensor-math peak is therefore the wrong ceiling for an operation that runs on another execution pipeline. They are also particular: they want specific data types, specific shapes and specific alignment, and the consequences of an unsupported configuration depend on the library or instruction: padding, a different kernel or an error are all possible. Inspect the selected kernel rather than assuming a silent fallback.

How nearby storage reduces data movement

The diagram separates storage from execution; it does not require every access to visit every cache. Some loads bypass L1; asynchronous copies can move global data into shared memory without staging it in thread registers. Capacities and latencies depend on the architecture and access pattern, so there is no fixed tenfold ratio between levels.

Each thread uses registers for fast access to its working values. The SM has a limited register budget, so a kernel that uses more registers per thread may fit fewer resident warps. That leaves fewer alternatives to run while a warp waits for data. More registers can still be useful if they let the kernel reuse values instead of loading them again.

Threads are also grouped into blocks. Threads in the same block can cooperate through shared memory, whose use the program manages explicitly. For example, they can load a small part of a matrix, called a tile, and reuse it for several calculations. Each reuse avoids another trip to larger, more distant memory.

L1 is an on-SM cache for applicable loads, distinct from explicitly managed shared memory even where they share physical capacity. L2 is shared across SMs. Locality affects both; cache policies and access hints can also affect behavior. Neither cache guarantees that an entire model remains resident.

High-bandwidth memory holds accelerator-resident weights and other large allocations. The previous lesson examined a case where reads from this memory dominate. Reuse in smaller storage reduces those reads; offloaded state needs a separate transfer budget.

The GPU Memory Hierarchy concept page has the sizes and latencies. What matters for infrastructure work is the shape: reuse can reduce traffic from large storage to the execution units.

Where this shows up in a real answer

For a large dense model whose participating weights exceed cache capacity, batch-one linear layers have little weight reuse across output tokens. Most weights must be fetched again for the next step. Small models may fit in cache, and MoE routing selects a subset of experts; state those assumptions before using a whole-model bandwidth estimate.

Add a batch and reuse appears: the same weight, read once, now serves many sequences. That is the mechanism behind batching, stated in the memory hierarchy rather than as a scheduling trick. It is also why the benefit flattens once you are compute-bound, which the next lesson makes precise.

Check whether more registers buy useful reuse

Occupancy describes how many warps are resident compared with the maximum the SM supports. To see how registers can limit it, consider an illustrative SM with 65,536 32-bit registers, a limit of 2,048 resident threads, and blocks of 256 threads. For this first estimate, ignore the hardware's allocation increments and other limits:

32 registers/thread × 256 threads/block = 8,192 registers/block
register-limited blocks = floor(65,536 / 8,192) = 8
resident threads = 8 × 256 = 2,048: 100% of the assumed thread limit

64 registers/thread × 256 threads/block = 16,384 registers/block
register-limited blocks = floor(65,536 / 16,384) = 4
resident threads = 4 × 256 = 1,024: 50% of the assumed thread limit
sanity: doubling registers/thread halves this register-limited residency ceiling

Actual occupancy also depends on shared memory, block limits and allocation granularity. Keeping a reused value in registers can save traffic even while residency falls. Prefer the extra registers when measured kernel time falls without harmful spills; reverse the change if lost latency hiding costs more than the saved loads. The CUDA execution and memory guide describes warp scheduling and the on-SM resource constraints behind this trade.

Do this before moving on

Take three questions and label each one execution or memory before answering:

1. A kernel runs at a third of the expected speed and the profiler shows most warps stalled waiting on data.
2. Half the threads in a warp take one branch of a conditional and half take the other.
3. Two sequences of the same length produce very different throughput when run at different batch sizes.

Then say which measurement would distinguish the causes. For the first, compare memory throughput and eligible warps: waiting on data can mean saturated bandwidth or insufficient independent work. For the second, inspect active lanes in the divergent region. For the third, compare weight traffic per emitted token. Finally, repeat the register calculation at 48 registers/thread: the simplified register limit admits five blocks, or 1,280 threads, before other constraints.

Go deeper

• GPU Execution Model covers threads, warps, blocks and scheduling in the depth this lesson only sketches.
• GPU Memory Hierarchy has the actual capacities and latencies at each level, which is what you need for sizing.
• Tensor Cores and Matrix Units explains what these units demand of your data before they will run fast, so you can inspect the selected instruction and operand path.
• What is a tensor core and what does it need? is the interview version, including the requirements a kernel has to satisfy.
• Warp divergence and why it costs you works through the masking cost with numbers attached.

Key takeaways

• Separate execution resources from storage, then explain how register and shared-memory budgets constrain residency.
• A 16/16 split across equal-cost paths uses approximately half the lanes in that region; whole-kernel slowdown needs measurement.
• Tensor cores are execution units that do a matrix multiply-accumulate per instruction, and they are particular about types, shapes and alignment.
• Reuse in registers, shared memory and caches can reduce traffic from high-bandwidth memory.
• In the illustrative register budget, 32 to 64 registers/thread reduces the register-limited ceiling from eight blocks to four.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1Where do tensor cores sit?

   On the host side, as part of the driver's scheduling logicBetween L2 cache and high-bandwidth memory, as a staging area for matricesInside each streaming multiprocessor, as execution units alongside the general arithmetic unitsIn the memory hierarchy, as a fast cache for matrix operands

2. 2Threads within one warp hit an if-else where half take each branch. What does the hardware do, and what does it cost?

   Answer it, then reveal

3. 3A kernel is stalled waiting on memory. An engineer proposes using more registers per thread to keep more data close by. What is the catch?

   Registers are shared across the whole SM, so this starves other kernelsThere is no catch; this is the standard fixRegister memory is slower than shared memory for this access patternIf registers limit residency, more per thread reduces the warps available to hide latency