Start with bytes when sizing low-batch decode

TL;DR: When a dense model generates one token at a time for a single request, moving its weights from memory can take longer than the arithmetic. With BF16 weights, the linear layers do about one floating-point operation per byte of weight traffic. Use that estimate to investigate memory bandwidth first, then check whether batching or lower precision would reduce the cost.

Where you are. This first foundations lesson explains why a GPU can spend much of its time moving data. You will estimate that cost and use it to choose a performance experiment.

Compare arithmetic with the bytes it needs

Imagine a model answering one request, generating a token at each step. Its weights are the stored numbers used in its calculations. In this example, the model is dense and its weights are too large for the GPU's small on-chip cache, so generating the next token requires reading the weights from larger memory again. Faster arithmetic helps only if the GPU can supply the data quickly enough.

To compare those two costs, suppose an accelerator can perform 1,000 trillion floating-point operations per second, written as 1,000 TFLOP/s, and move 2 trillion bytes per second from memory, or 2 TB/s. Dividing gives 500 operations per byte. To reach its arithmetic peak, the workload must do at least 500 operations for every byte it moves. Use the arithmetic rate for the data type and operation you actually run.

If the workload does fewer than 500 operations per byte, this accelerator reaches its memory-bandwidth limit before its arithmetic limit. Adding more arithmetic capacity alone cannot fix that. Above the ratio, arithmetic becomes the tighter limit. These are best-case limits, often drawn as a roofline; launching work, inefficient memory access and too little parallel work can make the actual result slower.

The name for operations-per-byte is arithmetic intensity, and the ratio the hardware imposes is the ridge point:

arithmetic intensity  I = FLOPs performed / bytes moved       (a property of your work)
ridge point           R = peak FLOP/s / memory bandwidth      (a property of the chip)

I < R  ->  bandwidth sets the lower theoretical ceiling
I > R  ->  arithmetic sets the lower theoretical ceiling
sanity: both I and R have units of FLOP/byte, not FLOP/s

These ratios narrow the experiments worth trying. The Roofline Model concept page derives the bound; here you will use it to choose a change and predict its effect.

Why this matters more for serving than for training

Training and prompt processing, called prefill, often use large matrix multiplications. They can reuse each weight across many tokens, doing more arithmetic for each byte read. That can move them above the ridge point, where arithmetic is the tighter limit. Other operations in the same job may still spend most of their time moving data or waiting. Check the operation you are investigating before describing the whole service as memory-bound or compute-bound.

For the linear layers of a dense model serving one sequence, each participating weight contributes roughly a multiplication and an addition: two floating-point operations, or FLOPs. The BF16 number format stores each weight in two bytes. That gives 2 FLOPs / 2 bytes = 1 FLOP/byte, far below our example accelerator's 500 FLOPs/byte ridge point.

This calculation covers weight traffic in the linear layers. To estimate the whole step, you must also account for attention, cached state, embedding lookups and runtime overhead. A mixture-of-experts model needs a different weight count because each token uses selected experts rather than every expert. The weights kept in memory still count toward the capacity needed to load that model.

Use this hypothetical 8-billion-weight linear-layer workload to check the units:

P = 8 × 10^9 weights; B = 1 token; q = 2 bytes/weight
linear work ≈ 2 × P × B = 16 × 10^9 FLOPs
weight traffic ≈ P × q = 16 × 10^9 bytes = 16 GB
weight-only intensity ≈ 2PB / Pq = 2B/q = 1 FLOP/byte
assumed sustained bandwidth = 2 × 10^12 bytes/s
weight-read time ≈ 16 × 10^9 / (2 × 10^12) = 8 ms
sanity: the weight-only ceiling is 1 / 0.008 = 125 tokens/s, before other work

These are illustrative inputs, not a measured model benchmark. At fixed batch size the parameter count cancels from intensity, but it does not cancel from latency.

At batch 16, the same ideal BF16 weight pass serves 16 tokens and reaches 16 FLOPs/byte. Real traffic also includes activations and KV reads, so linear growth eventually stops. NVIDIA’s matrix multiplication guide derives the full matrix traffic estimate and explains why dimensions affect achieved throughput.

The curve shows the shape of the bound, not measured performance. Batching moves weight-dominated work to the right. Quantization also increases intensity by reducing bytes; it does not increase physical memory bandwidth. A different arithmetic format can change the compute ceiling as well, so recompute the ridge point for that format. Caching helps only when the working set has reuse and fits the chosen cache.

What to do with this in an interview

For “why is GPU utilization at 90% while throughput is bad?”, first identify the metric. The nvidia-smi reference defines utilization.gpu as the fraction of the sample interval with at least one kernel executing. It does not measure achieved FLOPs or resident-warps occupancy. A kernel waiting on memory can keep this reading high.

So the sequence is:

1. Estimate the bytes the work must move and the operations it must perform.
2. Divide to get intensity; compare against the ridge point.
3. Establish whether memory traffic or arithmetic gives the tighter estimated limit.
4. Choose a change that reduces that cost and predict what it should improve.

Start with batching when weight traffic dominates and the latency budget permits more concurrent work. Reverse that choice when extra waiting violates time-to-first-token, or when KV traffic and attention dominate long-context decode. The interview follow-up is to predict the change: doubling batch should approximately double weight-only intensity, not necessarily request throughput.

Do this before moving on

Repeat the 8-billion-weight calculation at batch one for BF16 and 8-bit weights, then for BF16 at batch eight. State which traffic you omit. Your weight-only intensities should be 1, 2 and 8 FLOPs/byte respectively; with the assumed bandwidth, the corresponding weight-read floors are 8, 4 and 8 ms per step. Explain why the third step emits eight tokens across eight sequences without promising eight times the observed throughput.

Then look up the ridge point of any current accelerator by dividing its quoted peak arithmetic rate by its quoted memory bandwidth, and write down how far apart the two numbers are. Carry that gap into the next lesson.

Go deeper

• Roofline Model is the formal version of the sketch above, with the derivation and the shape of the curve worked out properly.
• Memory-Bound vs Compute-Bound Kernels takes the same idea down to individual kernels and shows how the classification changes what you optimize.
• GPU Memory Hierarchy explains where the bytes actually live, which is the next question once you know bandwidth is the constraint.
• Is LLM decode memory or compute bound? is this lesson as an interview question, with the arithmetic written out the way you would say it aloud.
• Why does batch size change the TFLOPS you achieve? follows the same reasoning to the answer people find surprising.

Key takeaways

• The ridge point compares arithmetic throughput with memory bandwidth using the same operation type and precision.
• Arithmetic intensity is FLOPs per byte for your work; the ridge point is peak FLOP/s over bandwidth for the chip. Compare them before optimizing.
• Dense BF16 linear layers at batch one have weight-only intensity near 1 FLOP/byte; the approximation is 2B/q for batch B and q bytes per weight.
• The illustrative 16 GB weight pass at 2 TB/s takes at least 8 ms; attention and other work increase token latency.
• GPU utilization measures time with kernels executing; it does not establish arithmetic efficiency or SM occupancy.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A dense BF16 service at batch one is measured to be limited by weight-read bandwidth. The team proposes moving to an accelerator with double the peak arithmetic throughput and the same memory bandwidth. What happens?

   Throughput roughly doubles, because the model is compute-boundThroughput halves, because the new chip has worse memory localityLatency improves but throughput does notThe weight-read ceiling stays unchanged; arithmetic was not the binding term

2. 2For dense linear layers, why does parameter count cancel from weight-only intensity, and what remains?

   Answer it, then reveal

3. 3A time-based kernel-activity counter reports 95% while throughput is low. What does that counter establish?

   That the arithmetic units are nearly saturatedThat memory bandwidth is nearly saturatedThat the batch size is too smallThat at least one kernel executed during 95% of the sample interval