TL;DR: Five stages with the bytes at each boundary: ingest 5 PB of raw crawl, filter to about 400 TB of candidate text, deduplicate to about 150 TB, tokenize to 15T tokens at 2 bytes each (30 TB of uint16 shards), and serve at the training rate of 1.7 M tokens per second, which is 3.4 MB/s and trivial. To finish in two weeks the ingest and filter stages read at about 4 GB/s, the dedup stage builds MinHash signatures for roughly 3 billion documents, and everything is a content-addressed, restartable map-reduce with a manifest so a stage can be re-run without re-running the ones before it.
How to approach it
Ask the token target, the sources (crawl, licensed corpora, code), the deadline to have the data ready, and the training rate it must feed. Say you will carry the byte count through every stage, because the shrink ratio is what sizes each stage. Draw the stages as a DAG with manifests between them. Size each stage from its bytes and the deadline, then take dedup and the serving format as the deep dives.
A strong answer
A typical situation: a lab is six weeks from a 10,240-GPU run that will consume 15T tokens over 100 days, and the data team has raw crawl, a licensed book and paper corpus, and a code corpus, and two weeks of a 2,000-core CPU cluster to prepare it. Evaluation and Data Pipeline Infrastructure describes the family of pipelines; the numbers below are for this one.
The bytes at each boundary, and the throughput to finish in two weeks (1.2 × 10⁶ s).
ingest: 5 PB of raw crawl (WARC with HTML, headers, boilerplate); read rate = 5e15 ÷ 1.2e6 s ≈ 4.2 GB/s
object storage at 100 MB/s per stream → 42 parallel readers minimum; use 400 for tail tolerance
extract + filter: HTML to text drops ~90% of bytes; language ID, a quality classifier, a toxicity filter
output ≈ 400 TB; CPU-bound: extraction at ~20 MB/s per core → 5e15 ÷ 2e7 = 2.5e8 core-seconds
≈ 70,000 core-hours → on 2,000 cores, 35 hours; the classifier adds about the same
dedup: exact (hash of normalized text) removes ~30%; near-duplicate (MinHash, 128 permutations, LSH bands)
over ~3 billion documents removes another ~30% → ~150 TB
signatures: 3e9 × 128 × 4 B = 1.5 TB, one shuffle by band; the connected-components step is the long pole
tokenize: 150 TB of text at ~4 bytes per token ≈ 37T candidate tokens; the mixture selects 15T
tokenizer at ~1 M tokens/s per core: 1.5e13 ÷ 1e6 = 1.5e7 core-seconds ≈ 4,200 core-hours, 2 hours on 2,000 cores
output: 15e12 × 2 B = 30 TB of uint16 token IDs, 1 GB shards → 30,000 shards
serve: 15T tokens over 100 days = 1.7 M tokens/s = 3.4 MB/s; per data-parallel rank of 160: 21 KB/s
sanity: the training run reads 30 TB in 100 days; the pipeline read 5 PB in 14 days; serving is
never the bottleneck, and every hour saved is in filtering and dedup
Dedup, the expensive stage. Exact dedup is a hash join. Near-duplicate dedup is MinHash: shingle each document, hash with 128 permutations, group into 16 bands of 8, and any two documents sharing a band are candidates. Candidates form a graph; connected components pick one survivor per cluster. The shuffle by band is a 1.5 TB sort, fine; the components step over billions of edges is what takes a day, and it must be checkpointed. Dedup also runs across sources, because a licensed book that also appears in the crawl should count once. The trade-off inside: a lower similarity threshold removes more near-duplicates and more legitimate variants (versions of a document); the answer is to dedup at 0.8 Jaccard and keep the cluster size as a feature so the mixture can downweight rather than delete.
The serving format. Shards are fixed-size token arrays with a manifest listing shard hash, token count, source and mixture weight. The loader is a deterministic function of (manifest, seed, step): every data-parallel rank can compute which tokens it reads at step n without coordination, which makes resume from a checkpoint exact and makes "which documents did the model see before the loss spike at step 41,000" answerable. Shuffling is two-level: shards are permuted per epoch by the seed, and a buffer of a few thousand documents is shuffled in memory per rank; a global shuffle of 30 TB is unnecessary because document order within a shard is already randomized at write time.
Restartability. Every stage writes content-addressed outputs and a manifest; a stage re-runs only the inputs whose hashes changed. A filter threshold change re-runs filter, dedup and tokenize on the affected sources and nothing else. Without this, every parameter change is a two-week rerun.
The trade-off to commit to: tokenize offline into uint16 shards rather than tokenizing in the loader. It costs 30 TB of storage and a full re-run when the tokenizer changes, and buys a loader that is a byte copy at 3.4 MB/s with zero CPU pressure on 1,280 training nodes. The reversal condition: a tokenizer still under development, or a curriculum that changes the mixture weekly, where online tokenization from the deduped text (150 TB, still small) keeps iteration fast at the cost of loader CPU. Dataset Lifecycle: Ingest, Shard and Retain is the same pipeline as a lifecycle, and p99 shard read latency is what starves the GPUs when it slips.
Failure modes to name: a filter that silently drops a language (per-source, per-language counts at every boundary, alarmed on change); dedup that removes the eval set's near-duplicates from training but not exact leakage (hash-match the eval prompts against the tokenized shards and report contamination per eval); a shard written twice under different hashes after a retry (idempotent writes keyed by input hash); a loader that reads a shard sequentially from one object and starves at the tail (prefetch two shards ahead); token counts in the manifest that disagree with the shards (verify on write).
What interviewers probe next
- "Where does the two weeks actually go?" Filtering and dedup, roughly 35 hours each of 2,000-core time plus the components step; ingest is bandwidth and tokenize is two hours.
- "How do you know the mixture you served is the mixture you designed?" The loader logs (source, tokens) per step; a nightly job sums them against the manifest weights.
- "What if a source has to be removed after training starts?" The manifest is versioned; the loader switches at a step boundary, and the change is recorded so the run's data lineage is reconstructible.
Common mistakes
- Quoting token counts with no bytes, so no stage can be sized.
- Treating dedup as a hash join and being surprised by the near-duplicate graph.
- A loader that needs a coordinator to decide what each rank reads, which makes resume inexact.
- Skipping contamination checks against the eval set.
Key takeaways
- 5 PB raw → ~400 TB filtered → ~150 TB deduped → 30 TB of uint16 tokens; carry the bytes.
- Two-week deadline: ingest at 4 GB/s, filter and dedup at about 35 hours each on 2,000 cores, tokenize in 2 hours.
- MinHash with 128 permutations in 16 bands; the components step is the long pole and must checkpoint.
- Serving is 3.4 MB/s; make the loader a deterministic function of (manifest, seed, step).