numerics

Use proportional bit layouts and adjacent values around one to distinguish range from precision. Then choose formats from numerical tests and actual kernel support.

Build a numerical reference, count the traffic and investigate a row-wise fused kernel. Separate bandwidth lower bounds from measured performance.

Watch a low-precision sum stop changing and a small weight update round away. Separate storage precision, accumulation, optimizer state and dynamic range before choosing a mixed-precision training policy.

Longer sequences exercise more state updates, different boundaries and sometimes different kernel paths. Use a two-tile example to check online softmax, then test masking, indexing and precision against a stable reference.

Derive row-major offsets, broadcast the bias correctly, and compare a lane-by-lane CPU reference with a vector expression. Explain when shared memory adds no reuse.

Distinguish stale telemetry, skipped updates and a real learning plateau using counters, parameter changes and a fixed evaluation batch.

Derive the scale, handle zero tensors, clamp before casting, and separate rounding error from clipping. Compare one outlier under tensor and channel scales.