deduplication

Exact duplicates are a hash and a group-by. Near-duplicates are a similarity search over billions of documents, which becomes a shuffle rather than a computation. The signature arithmetic, why the shuffle is the expensive stage, and where a GPU classifier fits in a pipeline that is otherwise all CPU.

Four mechanics do most of the work: symptom-based rules rather than cause-based, burn rate rather than thresholds, deduplication so one event is one page, and suppression during known windows. The arithmetic for each, and the rule that keeps the set from growing forever.

Fifteen trillion tokens starts as a few petabytes of raw text and ends as 30 TB of shards a training job reads at 1.7 million tokens per second. The stages, the bytes at each boundary, the throughput each one needs to finish in two weeks, and the two stages where the pipeline actually spends its time.