TL;DR: Fusion can remove intermediate tensor stores/loads and some launch boundaries. Three pointwise stages over an S-byte activation have 6S logical activation traffic when separate and 2S when fused, before side inputs and other obligations. That is a traffic reduction, not a guaranteed 3× speedup. Register pressure, lost parallelism, numerical changes and saved-for-backward state can reduce or reverse the benefit.
How to approach it
First follow an intermediate tensor: where the separate implementation writes it, where the next stage reads it, and how fusion removes those transfers. Count only the bytes the fusion actually removes. Then account for launch exposure, register use and the numerical contract. Finish by measuring the complete block, including any matrix multiplication that remains.
A strong answer
A model executes a matrix multiply followed by several small pointwise operations. The profile shows time between kernels and repeated activation traffic. Fusion is worth testing, but “fewer kernels” is not yet a performance explanation: the useful question is what work disappears and what new cost the fused implementation introduces.
Follow the intermediate, not just the kernel count
For a simple three-stage chain, suppose each stage consumes and produces one S-byte activation, and the intermediate outputs have no other consumers. The separate implementation writes two full intermediates that the next stages read. The fused implementation can keep their per-element working values in registers or other on-chip storage and write only the final result.
This mechanism is not “put the intermediates in L2.” It avoids materializing the full intermediate tensors at all, unless compilation, spills or semantics require otherwise. Kernel Fusion describes the transformation; GPU Memory Hierarchy separates registers, explicit shared memory and hardware caches.
Toy activation: 32 × 4,096 × 16,384 BF16 values
S = 4,294,967,296 bytes ≈ 4.295 GB
three separate activation reads and writes = 6S ≈ 25.770 GB
one fused activation read and write = 2S ≈ 8.590 GB
at hypothetical sustained 3 TB/s: traffic floors ≈ 8.590 ms and 2.863 ms
sanity: bias, masks, random-state work and saved tensors are not included in 6S or 2S.
A chain containing dropout may need a mask or a reproducible random-number mapping; training may require values for backward or a deliberate recomputation policy. Side inputs, dtype conversions and other consumers can prevent the simplified count from applying. Some separate-kernel traffic may hit cache instead of HBM, so logical activation bytes are not automatically physical HBM bytes.
A traffic gain is bounded by the rest of the work
Suppose an illustrative block takes 25 ms in a matrix multiply and 8.59 ms in the following chain. Its starting total is 33.59 ms. Reducing the chain to 2.863 ms gives 27.863 ms, or about 1.21× overall speedup, not 3×. Even removing all 8.59 ms without any added cost would give only 33.59/25 ≈ 1.34×.
A fused GEMM epilogue can avoid another output pass, but can also change register pressure, scheduling and the GEMM implementation selected. Do not claim the GEMM time must remain exactly unchanged. Equally, 70% of a tensor-core peak does not prove the remaining 30% is available to fusion or identify the bottleneck. Roofline Model helps frame traffic and compute bounds; measured dependencies and resource use decide the result.
Three useful shapes, different constraints
A pointwise chain can often compute each output independently. A reduction plus its consumers, such as softmax, must share a reduced value before producing outputs. A GEMM epilogue operates on accumulators before their final store. These are different opportunities, not a promise that any adjacent operations can be combined.
Triton's fused-softmax tutorial illustrates a row that can be handled within the kernel's supported resource budget. A very wide row may need tiling, multiple passes or another algorithm; “fused” does not guarantee one HBM read for every shape. FlashAttention Internals explains a more specialized case: attention avoids materializing the full score matrix while maintaining the reduction state needed for a correct result.
PyTorch's compiler documentation describes the compilation stack. Actual fusion depends on the captured graph, backend, supported operations and layout. Data-dependent behavior does not universally make fusion impossible, but host synchronization or unsupported graph regions can prevent a particular transformation. Inspect generated kernels rather than promising automatic fusion of every chain.
Exposed launch time is the quantity to save
Removing launches helps when their overhead or dependent scheduling gaps lie on the critical path. Summing all CPU API durations can double-count overlap with GPU work. Capture a timeline and separate CPU dispatch, GPU execution, gaps and synchronization. CUDA graphs can reduce repeated dispatch overhead while retaining multiple kernels, so graph replay and fusion solve overlapping but different problems.
At large sizes, extra live values can increase registers and cause local-memory spills. A reduction fused into fewer blocks can reduce parallelism. Resource pressure may turn a smaller logical byte count into more actual traffic or a longer critical path. Occupancy and Register Pressure gives the resource model; occupancy itself is a clue, not the objective.
Correctness also needs a defined tolerance. Fusion may remove an intermediate rounding step or alter reduction order, accumulation precision, NaN handling or random-number assignment. Compare the intended semantics on boundary values and representative inputs. For dropout, matching a seed alone does not prove the same element-to-random-number mapping after fusion.
Keep the fusion when the complete workload preserves the required outputs and improves the target latency or throughput. Split it again when spills, reduced parallelism, compilation overhead or numerical differences consume the gain. A profile dominated by an unrelated operation calls for optimizing that operation, even if another fusion is technically possible.
What interviewers probe next
- "Where did the memory go?" Removed intermediates stay as local working values rather than full tensors; verify spills and required saved state.
- "Can fusion help a compute-heavy block?" It can remove surrounding work, but the overall gain is bounded and the implementation may change.
- "Do graphs replace fusion?" No. Graph replay reduces dispatch overhead; fusion can also remove intermediate traffic.
Common mistakes
- Equating 6S→2S traffic with a guaranteed 3× runtime gain.
- Ignoring side inputs, backward requirements, cache reuse or spill traffic.
- Calling an operation universally bandwidth-bound just because it is pointwise or a reduction.
Key takeaways
- Three simple stages reduce logical activation traffic from 6S to 2S under the stated assumptions.
- The toy 25 ms + 8.59 ms block improves only about 1.21× when the chain reaches 2.863 ms.
- Verify generated code, numerical behavior and the entire workload before keeping a fusion.