methodology

Six things decide whether a kernel measurement means anything, and a script that ignores them can report a number that is wrong by a factor of two in either direction. What to control, which statistic to report, and the six ways a large speedup turns out to be a measurement artifact.

A single throughput number describes a configuration nobody runs. The four variables that have to be fixed, the curve that is the actual deliverable, and the bandwidth bound that turns the result from a description into a diagnosis.

The conversion is the easy part and the evaluation is the project. What calibration data does and why yours should come from production, the per-category evaluation that catches what an average hides, and the parts of the model that should not be quantized at all.