TL;DR: Build a versioned pipeline with measured yields at each boundary: ingest, extract/filter, deduplicate, tokenize, pack and serve. Choose token storage width from the actual maximum token ID. In the worked example a 128,000-ID vocabulary needs uint32, so 15T tokens occupy 60 TB of raw token payload. Average training consumption is small relative to raw ingestion, but loader stalls still depend on access patterns and tail latency.
How to approach it
Ask the source mix, usable-token target, tokenizer, completion deadline and training consumption rate. Carry bytes and records through the pipeline without assuming one universal bytes-per-token ratio. Explain global deduplication, immutable manifests and how a restarted loader resumes the intended sample stream.
A strong answer
Packing concatenates tokenized documents into fixed-length training sequences so no positions are wasted on padding. The pipeline produces valid-looking shards, but a token with ID 100,000 was written to a 16-bit integer. The storage estimate was attractive because it silently assumed a smaller vocabulary. Check the encoding before optimizing shard count.
A hypothetical vocabulary with IDs 0 through 127,999 exceeds uint16's maximum of 65,535. Uint32 can represent these IDs. Fifteen trillion tokens therefore need 15e12 × 4 = 60e12 bytes, or 60 decimal TB of token values. Indexes, offsets, provenance, replicas and object metadata are additional. A smaller tokenizer whose entire ID range fits uint16 could use 30 TB instead; never infer that choice from the word “token.” Packed formats require their own reader and CPU-cost analysis.
Planning example, decimal units:
Raw input 5 PB / 14 days = 4.13 GB/s sustained read floor
15T tokens / 100 days = 1.736 million consumed tokens/s
At uint32: 1.736 million × 4 = 6.94 MB/s average token payload
60 TB / 1 GB payload per shard = 60,000 shards before indexes/replicas
Sanity: reading raw source bytes and feeding packed tokens are different workloads.
Ingestion stores immutable source objects with origin, retrieval time, content digest and policy metadata. Extraction and quality filters emit derived records with transform versions and source links. Measure language/domain yields on representative samples before allocating the full fleet. If 400 TB of candidate text retains 70% after one filter and 70% after another, the result is 196 TB, not 150 TB. Those are hypothetical yields; actual retained tokens require tokenization of the selected content, not an invented constant bytes-per-token conversion.
Deduplication has both local and global work. Exact normalized hashes remove identical content under an explicit normalization policy. Near-duplicate methods generate candidates from shingle signatures and verify similarity before choosing retained records. Broder's document-resemblance paper explains the set-similarity and sampling foundation. A toy three-billion-document signature store with 128 uint32 values per document consumes 3e9 × 128 × 4 = 1.536 TB of raw signatures. Band indexes, shuffle traffic, candidate edges and provenance add to that. A connected component can contain A similar to B and B similar to C without A meeting the threshold against C; document the retention rule rather than calling every member pair a duplicate. Audit false removals by domain and language, and preserve the chosen survivor's source lineage.
A two-week five-PB read deadline implies 4.13 GB/s before retries or skew. At a hypothetical measured 100 MB/s per reader, 42 continuously productive readers provide the arithmetic floor. That does not size HTML extraction, quality classifiers or dedup shuffle; benchmark each stage's records/s, bytes/s and memory on its actual distribution. Keep bounded intermediate queues so a slow classifier cannot fill all disks. Separate policy filtering and held-out evaluation contamination checks from generic “quality” scores.
Use one recorded tokenizer version and special-token policy, and reject token IDs outside the storage format’s range. When packing documents into training sequences, retain their boundaries and the masks that control which positions can attend to each other and contribute to the loss. Write checksummed shards and indexes to immutable locations, and publish the manifest conditionally only when its dependencies are complete. Content hashes identify bytes; idempotent logical tasks additionally need input identity plus transform/config version and a commit rule. A global dedup or mixture change can invalidate outputs from many sources, even if only one source changed.
The loader consumes a manifest and a defined sample-order policy with sharding across data-parallel workers. Tensor-parallel ranks should not accidentally multiply logical sample consumption. Track dataset version, global sample position, sampler/RNG state, packing cursor and any state required by buffering. A seed and training step alone do not prove identical resume behavior after worker-count changes. Hugging Face's dataset comparison distinguishes exact indexed shuffling from buffered iterable shuffling and documents checkpointed iterator state. Verify the chosen loader's treatment of buffer contents and repartitioning with an actual stop/resume comparison.
The 6.94 MB/s figure is average unique token payload. Range overfetch, replicated reads, small random objects, synchronized refills and a slow rank can still stall training. Watch input-wait time on the step critical path, per-rank batch readiness, cache hit rate, p99 object-read latency and actual bytes per consumed token. Evaluation and Data Pipeline Infrastructure connects the artifacts; Stragglers and Hangs explains why one delayed rank can hold up the next step. I would start with immutable indexed shards and read-ahead, then change shard size or locality when traces identify the bottleneck. Do not rewrite the pipeline merely because a bandwidth average looks small.
Retain source-to-shard lineage for corrections and removals. Removing a source from the next manifest does not undo its influence on an already trained checkpoint; record which runs consumed which versions. Release a dataset after range checks, counts, checksums, yield audits and restart tests pass.
What interviewers probe next
- How do you resume with a different DP size? Specify global sample-order and repartitioning semantics, then compare consumed sample IDs through a stop/resume test. A seed alone is insufficient evidence.
- What does source removal change? Rebuild affected dataset versions through lineage and identify runs that consumed earlier versions. Removing future input does not erase previous training influence.
Common mistakes
- Choosing uint16 without checking the maximum token ID, including special tokens.
- Inferring loader reliability from average payload bandwidth or treating transitive near-duplicate groups as pairwise equivalents.
Key takeaways
- A 128,000-ID vocabulary cannot fit uint16; 15T uint32 tokens need 60 TB of payload.
- Five PB in fourteen days requires 4.13 GB/s sustained reads before processing overhead.
- 6.94 MB/s average consumption does not bound burst reads or per-rank loader stalls.
- Manifests establish lineage; sampler and loader state establish reproducible consumption.