load balance

A trillion parameters with 32 billion active per token: the weights need a rack, the compute needs a fraction of it, and the all-to-all between them decides everything. The memory plan across 72 GPUs, the decode step from bandwidth and two all-to-alls, the compressed attention cache, and what imbalance costs.

A plain hash ring with one point per replica measured a 47-fold load imbalance across eight replicas. With 150 virtual nodes each it fell to 1.19-fold, and adding a ninth replica moved 11.9 percent of keys against an ideal of 11.1. Both numbers, and what they cost.