roofline

Count two matrix shapes to derive the H100 ridge, throughput ceilings and time floors. Compare an illustrative measurement with its bound before deciding what to optimize.

Derive the weight-only shortcut, then add KV traffic and feasible eight-GPU placement. A worked latency floor shows why batch size alone cannot establish the decode bottleneck.

Derive HBM bandwidth from interface width and pin rate, then connect it to a model that actually fits. Separate raw interface capacity from measured decode time.

Use a profiler summary to form a memory-traffic hypothesis, then check counters, timing and useful work before choosing an optimization.

Derive how reusing weights changes arithmetic intensity, then separate the roofline ceiling from tile-wave effects and measured GEMM throughput.

Follow intermediate tensors through separate and fused stages, calculate the traffic and whole-block bounds, and check spills and numerical behavior.

Separate cache activity from DRAM traffic, check the access width and active lanes, and use a controlled layout change to test a profiler hypothesis. A few percentages cannot predict the fixed kernel’s runtime.

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Turn FLOPs and HBM bytes into resource bounds, then compare them with elapsed time. A tighter memory bound suggests an investigation rather than proving the measured bottleneck.

Derive the batch-295 shortcut, count the tensors it omits, and test whether the resulting kernel estimate is even inside the replica’s memory budget.