benchmarking

Peak numbers are almost never the reason a part wins or loses, and the software stack decides more deployments than the silicon does. The four things to measure, the question that eliminates most challengers in one conversation, and the pilot that gives a defensible answer.

Six things decide whether a kernel measurement means anything, and a script that ignores them can report a number that is wrong by a factor of two in either direction. What to control, which statistic to report, and the six ways a large speedup turns out to be a measurement artifact.

The one utilization number that cannot be gamed by recompute: model FLOPs over hardware peak, derived from a step time in four lines, with the band frontier labs land in and an itemized list of where the other 60% goes.

A single throughput number describes a configuration nobody runs. The four variables that have to be fixed, the curve that is the actual deliverable, and the bandwidth bound that turns the result from a description into a diagnosis.

A benchmark that replays one prompt measures the cache and reports a number production will never see. What the hit rate actually depends on, the eviction behaviour that erodes it under load, and the measurement that predicts the real gain.

Engine upgrades change defaults, kernel selection and scheduling at once, so attribution is the whole problem. The bisection that works on a version rather than a commit, the four categories of cause, and the pre-upgrade baseline without which none of this is possible.

Published data gives an interruption rate, and replacements are a fraction of interruptions, so the comparison needs care before it means anything. The derivation that turns one into the other, the answer for a healthy fleet, and the four cuts through the data that turn an anomalous number into a cause.

The tiling is ten lines and the theory is a working-set calculation. The interesting part is that the obvious benchmark does not measure what it appears to, because the library underneath is already tiled, and knowing which effect your numbers contain is the skill being tested.

A speedup number with no method behind it is worth nothing, and experienced interviewers stop listening at the number and start asking about the baseline. What makes a performance claim survive scrutiny, and the four ways a real one turns out to be smaller than it looked.

Naming three newsletters answers nothing. The answer that works has a filter, a source of ground truth that is not a blog post, and one thing you reproduced yourself, because that is the only evidence that you learn rather than accumulate.