batch processing

A tokenizer moves about a megabyte of text per second per core, which makes this a seven-hundred-core-day batch job rather than something to run during training. The throughput arithmetic per stage, the shard format the loader needs, and the determinism requirements that let you resume without corrupting a run.

Exact duplicates are a hash and a group-by. Near-duplicates are a similarity search over billions of documents, which becomes a shuffle rather than a computation. The signature arithmetic, why the shuffle is the expensive stage, and where a GPU classifier fits in a pipeline that is otherwise all CPU.