Tensor cores are not a kind of memory, and the distinction decides your answer
A GPU has two hierarchies that get confused with each other: an execution hierarchy of threads, warps and streaming multiprocessors, and a memory hierarchy of registers, shared memory, cache and high-bandwidth memory. Almost every muddled GPU answer is these two collapsed into one.
13 MIN
TL;DR: A GPU has an execution hierarchy (threads grouped into warps, scheduled onto streaming multiprocessors) and a memory hierarchy (registers, shared memory, L2, high-bandwidth memory), and they are different things. Tensor cores are execution units that do a small matrix multiply per instruction. Knowing which hierarchy a question is about is most of answering it.
Where you are. Second lesson of the trunk course. The previous lesson established that bandwidth is the constraint; this one says where the bytes actually live and what consumes them, which is what you need before any sizing arithmetic.
Two hierarchies, routinely collapsed into one
The single most common muddle in a GPU interview is treating "tensor core" as a place data sits. It is not. It is an execution unit, and it lives in the other hierarchy entirely.
| Execution hierarchy | Memory hierarchy | |
|---|---|---|
| What it is | What runs the work | Where operands live |
| Units | thread → warp → block → streaming multiprocessor | registers → shared memory → L2 → high-bandwidth memory |
| Scales with | how much parallel work you expose | how much data you touch and how you touch it |
| Tensor cores live here | yes, as specialised units inside each SM | no |
| The question it answers | "is there enough work in flight?" | "how far did each byte travel?" |
Say a question out loud and it will belong to one of the two columns. "Why is occupancy low" is execution. "Why is this kernel reading four times more than it needs to" is memory. Answers that mix the columns are the ones that do not survive a follow-up.
The execution side, in the detail that matters
Work is issued in warps: a fixed-size group of threads, conventionally 32, that execute together. This is the unit that matters, and it has one consequence worth memorising: if threads inside a warp take different branches of an if, the warp executes both sides with the inactive threads masked off. That is divergence, and it costs you exactly the fraction of work that was masked.
Warps are scheduled onto streaming multiprocessors, each with its own register file, its own slice of shared memory and its own execution units. An SM keeps many warps resident at once so that when one stalls on a memory read, another can issue. That is the whole trick of the architecture: it does not avoid memory latency, it hides it behind other work.
Tensor cores sit inside the SM alongside the general arithmetic units. Where a general unit multiplies two numbers, a tensor core performs a small matrix multiply-accumulate as a single instruction. Nearly all the headline arithmetic throughput of a modern accelerator comes from these units, which means a workload that cannot use them is running on a small fraction of the chip. They are also particular: they want specific data types, specific shapes and specific alignment, and a kernel that misses those requirements silently falls back to the slow path.
The memory side, and the only ordering worth memorising
Four levels, and the property that matters is that each step outward is roughly an order of magnitude larger and an order of magnitude slower.
Registers are per thread, the fastest storage, and scarce. Using more registers per thread means fewer threads fit on an SM, which reduces how much work is available to hide latency. This is a real trade rather than a bug.
Shared memory is per block and manually managed. It is the level that makes tiling work: load a tile once from far away, then reuse it many times from close by. Almost every fast kernel is organised around this idea.
L2 is chip-wide and automatic. You influence it by access pattern, not by instruction.
High-bandwidth memory is where the model weights live, it is measured in gigabytes, and it is the thing the previous lesson identified as the constraint. Everything above exists to avoid going here.
The GPU Memory Hierarchy concept page has the sizes and latencies. What matters for infrastructure work is the shape: fast is small, and reuse is the only way to make a large working set behave like a small one.
Where this shows up in a real answer
A model's weights are gigabytes, so they live in high-bandwidth memory and there is nowhere else for them. Decoding one token reads all of them once. No amount of cleverness about registers or shared memory changes that, because there is no reuse to exploit within a single token.
Add a batch and reuse appears: the same weight, read once, now serves many sequences. That is the mechanism behind batching, stated in the memory hierarchy rather than as a scheduling trick. It is also why the benefit flattens once you are compute-bound, which the next lesson makes precise.
Do this before moving on
Take three questions and label each one execution or memory before answering:
- A kernel runs at a third of the expected speed and the profiler shows most warps stalled waiting on data.
- Half the threads in a warp take one branch of a conditional and half take the other.
- Two sequences of the same length produce very different throughput when run at different batch sizes.
Then say which hierarchy each fix belongs to. The first two have clean answers; the third is deliberately the one that spans both, and noticing that is the point.
Go deeper
- GPU Execution Model covers threads, warps, blocks and scheduling in the depth this lesson only sketches.
- GPU Memory Hierarchy has the actual capacities and latencies at each level, which is what you need for sizing.
- Tensor Cores and Matrix Units explains what these units demand of your data before they will run fast, which is where most silent slowdowns come from.
- What is a tensor core and what does it need? is the interview version, including the requirements a kernel has to satisfy.
- Warp divergence and why it costs you works through the masking cost with numbers attached.
Key takeaways
- Execution and memory are two separate hierarchies; most confused GPU answers collapse them into one.
- A warp is the unit of execution, and divergence inside one costs you the masked fraction of the work.
- Tensor cores are execution units that do a matrix multiply-accumulate per instruction, and they are particular about types, shapes and alignment.
- Fast memory is small: reuse is the only way to make a large working set behave like a small one.
- Model weights live in high-bandwidth memory with no reuse within a single token, which is exactly why batching exists.
Check yourself
Answer before you look. Recalling it is what makes it stick; recognising it does not.
1Where do tensor cores sit?
2Threads within one warp hit an if-else where half take each branch. What does the hardware do, and what does it cost?
3A kernel is stalled waiting on memory. An engineer proposes using more registers per thread to keep more data close by. What is the catch?
Sign in to track which lessons you have finished.