Sharding is what makes a large model fit, and it is not free

Splitting the persistent state across ranks is what lets a large model train at all, and it converts a memory problem into a communication one. The stages differ in what they shard and what they therefore have to fetch back, and the transient peak is what actually decides whether it fits.

15 MIN · PREMIUM