tokenization

Measure batch readiness and producer capacity, then size prefetch for bursts without hiding a sustained throughput deficit.

Plan a reproducible tokenization build: measure CPU demand, choose a safe token dtype, publish complete shards and distinguish token lookup from training resume.

A 128,000-ID vocabulary breaks the 16-bit storage estimate. Follow the bytes from raw ingestion to 60 TB of token payload, then design deduplication, manifests and a restartable loader.

Decode fragmented UTF-8 with an incremental codec. Test malformed bytes, truncated endings and the boundary between byte decoding and event parsing.

Apply a learned merge order without retraining on the input. Test overlapping pairs, preserve UTF-8 bytes, and turn tokenizer differences into a capacity estimate.