GPU Execution Model

TL;DR: CUDA launches blocks of threads; an SM runs their warps of 32 threads. Keeping several warps ready lets useful instructions issue while another warp waits. Registers, shared memory and hardware limits bound that residency. GPUs also use caches. More resident warps help only when they expose work the machine can use; occupancy alone does not measure performance.

Why a model uses both a CPU and a GPU

A CPU runs general-purpose software and coordinates the application. A GPU provides many execution resources for parallel work, such as applying the same matrix operation across token representations. The CPU usually prepares inputs, allocates memory and launches GPU functions called kernels. A GPU does not replace the operating system or make every task faster; small or serial work can cost more to transfer and launch than it saves in computation.

The GPU also has device memory, often called VRAM. Its capacity tells you how much data can stay there; its bandwidth tells you how quickly bytes can move through a specified memory interface. HBM and GDDR are device-memory families. Registers and shared memory are smaller working stores closer to execution. GPU Memory Hierarchy follows that separate storage hierarchy. Tensors, Shapes and Dtypes explains the arrays those stores hold.

Follow one block onto the GPU

Imagine a kernel that processes an image, one thread per pixel. Increasing the block size makes it slower. The useful question is which resource changed: register allocation, shared memory, the number of resident blocks, or the work each memory transaction performs.

A grid is the collection of blocks in one kernel launch. A block is a cooperating group assigned to an SM, a streaming multiprocessor. The SM partitions its threads into warps. For a 256-thread block, that is eight warps; a partially filled final warp still occupies a warp slot. Different blocks can run concurrently or at different times. Ordinary kernels must not assume block-index order or that every block is resident together.

The CUDA Programming Model develops the launch and synchronization rules. This page explains the performance consequence: a resident warp can still be waiting, and a ready warp still needs a suitable issue slot.

Ready work hides waiting

Resident warps retain execution state on the SM, so selecting another eligible warp does not require an operating-system-style save and restore. That is the useful meaning of inexpensive switching. It does not mean unlimited issue bandwidth, zero dependency latency, or that every stalled warp has a ready replacement.

The diagram deliberately leaves empty slots. Four warps cannot necessarily cover a long memory wait. Instruction-level parallelism within a warp, other resident warps, asynchronous copies and reuse through caches can all help. If every warp waits at the same barrier or on a dependent load, a high resident-warp count may still leave execution units idle.

The GPU Memory Hierarchy matters here: registers, shared memory, L1 and L2 caches can reduce traffic or supply data sooner. Latency hiding and caching complement each other. Neither removes the cost of repeatedly fetching bytes that must reach HBM.

Calculate a residency bound

For an H100 SM, the documented limits include 65,536 32-bit registers and 64 resident warps. Use a 256-thread block with no shared-memory restriction for this simplified register calculation:

64 registers/thread × 256 threads = 16,384 registers/block
floor(65,536 / 16,384) = 4 blocks
4 × 256 / 32 = 32 resident warps
32 / 64 = 50% theoretical occupancy

At 32 registers/thread: 8 blocks → 64 warps → 100%
Sanity: halving this resource demand doubles the register-only bound.

These are bounds, not promised measured occupancy. Allocation granularity, block limits, shared memory and the grid size can lower the result. Use the compiler's resource report and occupancy tools for the exact kernel. The Hopper tuning guide lists these hardware limits.

Reducing register use can also cause spills into memory or remove useful work kept in registers. If the 50%-occupancy version already saturates its limiting pipeline, the 100% version may gain nothing and can lose performance. Occupancy and Register Pressure follows that trade further.

Divergence needs a precise explanation

A warp instruction operates on its active lanes. When threads take different paths, execution may issue separate instructions for those paths with different masks. The cost depends on path lengths, predication, reconvergence and available work; it is not always a twofold slowdown.

A warp groups logical threads, not a fixed set of 32 physical cores. A block with 33 threads uses two CUDA warps, with only one participating thread in the second. That partial final warp is different from threads taking different control-flow paths. Padding to a multiple of 32 does not by itself remove branch divergence or improve useful work.

The CUDA execution guidance describes independent thread scheduling on modern CUDA hardware, so do not use assumed permanent lockstep as a synchronization primitive. Communicating threads still need the appropriate synchronization and memory ordering. Threads choosing the same path can also issue inefficient scattered loads; Memory Coalescing is a separate concern.

Block synchronization does not create an ordinary grid-wide barrier. A kernel boundary with the appropriate stream ordering is one way to separate grid phases. Cooperative launches and thread-block clusters provide additional synchronization scopes under their own requirements. “Blocks can never synchronize” is therefore too broad, while a hand-built barrier that waits for unscheduled blocks can deadlock.

What interviewers are listening for

Explain the hierarchy, then connect it to a decision. For a slow kernel, inspect launch size, register/shared-memory limits, eligible warps, issue stalls and achieved bandwidth or compute throughput. A small grid may expose too little parallel work, but its thread count alone does not prove a particular percentage of the chip is idle.

Try increasing independent work when traces show insufficient eligible warps. Keep the change only if useful throughput improves without unacceptable spills or synchronization cost. Reverse it when the original kernel already saturates the relevant resource. The Roofline Model helps choose that resource; occupancy is supporting evidence, not the objective.

Key takeaways

• A 256-thread block contains eight CUDA warps; residency and readiness are different.
• The H100 register-only example gives 32 warps at 64 registers/thread, before other limits.
• Resident state makes warp selection cheap; caches and data reuse also matter.
• Use explicit synchronization instead of assumed lockstep or block order.
• Judge occupancy changes by measured useful work and the new limiting resource.