Find a performance limit before choosing a fix

TL;DR: Estimate the minimum time needed to move the data and perform the arithmetic, then compare it with the time you measure. Use the gap to choose an experiment, predict which part should get faster, and measure again. The same symptom can have several causes.

Where you are. You have compared arithmetic with memory bandwidth and followed data through the GPU. Now use those ideas to decide which performance change is worth trying.

Five useful investigation paths

When a model runs slowly, increasing the batch size or choosing a faster GPU may seem like an obvious next step. Either could help, but only if it changes the cost that is holding up the request. Start with a hypothesis about that cost and a measurement that could show whether you are right.

The table gives five useful starting points. It is not exhaustive: synchronization, storage, thermal throttling and competition for the scheduler can require a more specific investigation.

The experiment should preserve the variables you are not testing. A precision change that selects a different kernel does not isolate memory bandwidth, and adding ranks may alter placement as well as communication volume.

The procedure, without a profiler

You can form an initial estimate even without a profiler. Work through four steps, then use measurements to check the assumptions.

First, estimate a lower bound. Count the bytes the work must move and the operations it must perform. Divide bytes by the assumed available bandwidth to get a memory time, and operations by the assumed achievable arithmetic rate to get a compute time. The larger gives an optimistic minimum, or floor: even with perfect overlap, the work cannot finish before both requirements are met.

Second, compare the floor with the observation. If measured time is close to a valid floor, implementation tuning has limited headroom. Check that the byte count is necessary and the bandwidth assumption is independent of the slow run. Redundant work or degraded clocks can make a faulty implementation look close to an incorrectly chosen floor.

Third, investigate the gap. If the measured time is much longer than the floor, change one condition and see what happens. If adding devices makes the job slower, inspect communication and whether some devices reach the next step later than others. If a larger batch takes less than proportionally longer, weight reuse or shared launch overhead may explain the saving.

A lower weight precision that produces no speedup also deserves a closer look. The request may spend its time reading cached attention state, converting values or running a different kernel. The result alone does not rule out a memory bottleneck.

Finally, choose a change and predict its effect. State which measured interval should shrink and why. After the experiment, compare the result with that prediction. If they disagree, revisit the explanation before making more changes.

Why the estimate and the measurements both matter

Choosing a fix before estimating the costs can send you toward a part of the system that is already close to its limit. For a hypothetical example, imagine a collective communication operation (a step in which every participating GPU exchanges data, such as summing gradients) taking eight seconds. Calculate how many bytes each participating process must transfer and compare that with the usable link bandwidth. If the estimate is already near eight seconds, inspect unnecessary transfers and placement before expecting a kernel change to halve the runtime.

Dashboard values need the same care. Time-based GPU utilization reports intervals with kernels executing, while memory-used reports how much memory is allocated. Neither tells you how quickly useful work finishes. To find the critical path, follow the sequence of work and waits that determines completion time.

Work one case all the way through

Suppose a kernel must read and write 24 GB in total and perform 12 TFLOPs. Use hypothetical ceilings of 2 TB/s sustained memory bandwidth and 600 TFLOP/s for the operation type:

memory floor = 24 GB / (2,000 GB/s) = 0.012 s = 12 ms
compute floor = 12 TFLOPs / (600 TFLOP/s) = 0.020 s = 20 ms
optimistic overlapped floor = max(12, 20) = 20 ms
observed kernel time = 50 ms
maximum speedup from reaching this floor = 50 / 20 = 2.5×
if traffic halves: memory floor = 12 GB / (2,000 GB/s) = 6 ms
new overlapped floor = max(6, 20) = 20 ms
sanity: reducing bytes alone does not lower the arithmetic ceiling in this example

The 30 ms gap does not identify its cause. Inspect instruction mix, launch shape and dependencies before promising a 2.5× gain. The maximum assumes the bounds are attainable together. Sequential phases require a critical-path sum, while overlapping phases require accounting for shared resources.

Start by profiling the compute-heavy kernel in this example. Reverse that priority if the 50 ms measurement includes a 25 ms input wait outside the kernel: eliminating the wait may be the larger opportunity. NVIDIA’s GPU performance guide explains why latency and insufficient parallelism can keep execution below both arithmetic and memory ceilings.

Do this before moving on

Choose a workload you understand and use the five-row table to write what you would expect to observe for each possible limit. Pick the explanation you think fits best and make a prediction, such as which interval would shrink if you reduced weight precision. If you cannot predict the direction of the change, identify the missing measurement. An incorrect prediction is useful when it tells you which assumption to revisit.

Go deeper

• Memory-Bound vs Compute-Bound Kernels takes the first two of the five candidates down to kernel level, where the classification gets sharper.
• Roofline Model is the tool that separates the first two, and the notation the rest of the course uses.
• Arithmetic Intensity by Operation tabulates where common operations land, which is the shortcut once you trust the method.
• Your training run is slow: compute, memory, network or IO? is this procedure applied to the hardest version of the question.
• Read this Nsight summary: what is the bottleneck? is what the same reasoning looks like when you do have a profiler in front of you.

Key takeaways

• Use memory, arithmetic, latency, communication and input delivery as hypotheses; confirm the critical path with a controlled experiment.
• In the worked example, max(12 ms memory, 20 ms compute) gives a 20 ms optimistic floor against 50 ms observed.
• Halving traffic in that example reduces the memory floor to 6 ms but leaves the combined floor at 20 ms.
• Attach a prediction to the fix; that is what makes it a diagnosis rather than a guess.
• Time-based utilization and allocated memory cannot establish traffic efficiency; inspect the metric definition and timing scope.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A distributed job's step time grows noticeably as devices are added, while the work assigned to each device stays constant. Which hypothesis should you investigate first?

   Memory bandwidthCommunicationThe hostArithmetic throughput

2. 2A collective operation takes eight seconds and the team suspects a network fault. What is the first thing to compute, and how does it guide the investigation?

   Answer it, then reveal

3. 3The device shows regular short bursts of activity separated by consistent gaps. Which two candidates does that pattern point at, and how do you separate them?

   Communication and the host; separate them by device countMemory and arithmetic; separate them by comparing achieved bandwidth with achieved FLOPsLatency and the host; separate them by asking whether the gaps shrink when you enlarge the unit of work or when you feed the device fasterArithmetic and communication; separate them by lowering precision