cost per token

Put the weights in on-chip SRAM and the HBM wall disappears: tens of terabytes per second per chip, and a batch-1 step limited by the pipeline rather than the memory. The price is capacity: a 70B needs hundreds of chips per replica, and the cost per token depends on keeping every one of them busy.

Decode is bandwidth-bound, so cost per token tracks dollars per terabyte per second, and on spec the MI300X wins by 2x. The table, the memory-capacity term that changes the replica shape, and the software-efficiency discount that decides whether the spec ranking survives a benchmark.