A GPU is a bandwidth machine that happens to do arithmetic

TL;DR: A modern accelerator can do roughly two orders of magnitude more arithmetic per second than it can feed itself with operands. That imbalance, not the core count, is the single fact that explains why inference is slow, why batching works, why quantisation helps, and why half the optimisations in this field exist at all.

Where you are. First lesson of the trunk course. Everything later in this course, and in every specialisation, appeals back to the imbalance described here, so it is worth getting exactly right rather than approximately right.

The number everyone quotes is the wrong one

Ask what makes a GPU fast and you will be told it has thousands of cores. It does, and that fact will not help you answer a single infrastructure question.

Here is the problem with it. A processor is only as fast as its ability to get operands to the arithmetic units. A data-centre accelerator today can perform on the order of a thousand trillion floating-point operations per second, and read from its own memory on the order of a few trillion bytes per second. Divide one by the other and you get a ratio in the hundreds: for every byte the chip can read, it can perform hundreds of arithmetic operations.

That ratio is the whole subject. If the work you hand it does fewer operations per byte than the ratio, the arithmetic units sit idle waiting for memory, and buying a faster chip does very little. If the work does more, the memory system keeps up and the arithmetic is the limit.

The name for operations-per-byte is arithmetic intensity, and the ratio the hardware imposes is the ridge point:

arithmetic intensity  I = FLOPs performed / bytes moved       (a property of your work)
ridge point           R = peak FLOP/s / memory bandwidth      (a property of the chip)

I < R  ->  memory-bound: the arithmetic units wait. Faster maths does not help.
I > R  ->  compute-bound: memory keeps up. Now the arithmetic is the limit.

Two divisions, and they decide which optimisations are worth your time. The Roofline Model concept page derives this properly and draws it; this lesson exists to make sure you reach for it before anything else.

Why this matters more for serving than for training

Training and serving sit on opposite sides of the ridge point, which is why advice from one is often wrong for the other.

Training processes many tokens at once. Every weight it reads gets used against a large batch of data, so the operations-per-byte is high and the work lands on the compute-bound side. That is why training discussions are about FLOPs and utilisation.

Generating one token for one user is the opposite. To produce a single next token, the machine reads every weight in the model and does roughly two arithmetic operations per weight. The intensity is about 2, against a ridge point in the hundreds. It is memory-bound by two orders of magnitude, and it is not close.

Read the picture once and the field rearranges itself. Batching moves work to the right by amortising one weight read across more tokens. Quantisation moves the ceiling up by making each weight fewer bytes. Caching avoids re-reading. They are all the same move.

What to do with this in an interview

The question "why is my GPU at 90% utilisation but my throughput is bad" has one honest first answer, and it is not a list of guesses. Utilisation reports that a kernel was resident, not that it was doing useful arithmetic. A machine reading memory at full speed and computing almost nothing shows as busy.

So the sequence is:

1. Estimate the bytes the work must move and the operations it must perform.
2. Divide to get intensity; compare against the ridge point.
3. Name the side you are on before proposing a fix.
4. Choose an optimisation that moves the term that binds.

A candidate who says "it is memory-bound, here is the intensity, so more arithmetic throughput buys nothing and I would look at data movement" has answered. A candidate who says "increase the batch size" has guessed, and will be asked why, and will not have an answer.

Do this before moving on

Take any model you can name and compute its decode intensity from first principles. Count the bytes: the parameter count times the bytes per parameter is what a single decode step reads. Count the operations: roughly two per parameter. Divide. You should land near 2 operations per byte regardless of the model, because both terms scale with the parameter count and it cancels.

Then look up the ridge point of any current accelerator by dividing its quoted peak arithmetic rate by its quoted memory bandwidth, and write down how far apart the two numbers are. Carry that gap into the next lesson.

Go deeper

• Roofline Model is the formal version of the sketch above, with the derivation and the shape of the curve worked out properly.
• Memory-Bound vs Compute-Bound Kernels takes the same idea down to individual kernels and shows how the classification changes what you optimise.
• GPU Memory Hierarchy explains where the bytes actually live, which is the next question once you know bandwidth is the constraint.
• Is LLM decode memory or compute bound? is this lesson as an interview question, with the arithmetic written out the way you would say it aloud.
• Why does batch size change the TFLOPS you achieve? follows the same reasoning to the answer people find surprising.

Key takeaways

• A data-centre accelerator can do hundreds of arithmetic operations for every byte it reads; that imbalance explains most of this field.
• Arithmetic intensity is FLOPs per byte for your work; the ridge point is peak FLOP/s over bandwidth for the chip. Compare them before optimising.
• Generating one token for one user has an intensity near 2, which is memory-bound by two orders of magnitude.
• Batching, quantisation and caching are three ways of making the same move against the same constraint.
• Utilisation says a kernel was resident, not that it was doing useful work.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A service generates tokens for a single user at a time and is slow. The team proposes moving to an accelerator with double the peak arithmetic throughput and the same memory bandwidth. What happens?

   Throughput halves, because the new chip has worse memory localityLatency improves but throughput does notThroughput barely changes, because single-stream decode is memory-boundThroughput roughly doubles, because the model is compute-bound

2. 2Why does the arithmetic intensity of decode come out near 2 no matter how large the model is?

   Answer it, then reveal

3. 3A dashboard shows the accelerator at 95% utilisation while throughput is well below what the hardware should deliver. What does the utilisation figure actually tell you?

   That a kernel was resident on the device, and nothing about whether it was doing useful arithmeticThat the batch size is too smallThat memory bandwidth is nearly saturatedThat the arithmetic units are nearly saturated