Dataset Lifecycle: Ingest, Shard and Retain
A training dataset is not a file, it is a pipeline with four stages and a retention policy, and each stage has a different bottleneck. Ingest is metadata-bound rather than bandwidth-bound. Tokenization is CPU work that should happen once offline rather than every epoch. Sharding decides whether the training read is a stream or a storm of small files. And retention decides how much of the bill is paid for bytes nobody reads.
TL;DR: Four stages with four different bottlenecks. Ingest moves raw data in and is limited by operations per second on the metadata layer rather than by bytes, because raw corpora arrive as enormous numbers of small objects. Processing tokenizes and filters, which is CPU work at roughly a megabyte of text per second per core, so it belongs in a one-off batch job rather than on the training cluster's CPUs every epoch. Sharding writes the result as large sequential files, typically hundreds of megabytes each, which turns the training read from a metadata storm into a stream and is the single decision that most affects loader performance. Retention decides which tiers hold what for how long, and it is where most of the storage bill is either spent or saved. The recurring mistake is treating this as a storage capacity question when three of the four stages are bound by something else.
The four stages and their bottlenecks
| Stage | What it does | Bound by | The mistake |
|---|---|---|---|
| Ingest | Raw data lands in object storage | Metadata operations per second, not bandwidth | Sizing the tier from terabytes and discovering the file count |
| Process | Deduplicate, filter, tokenize | CPU, at roughly 1 MB of text per second per core | Running it on the training cluster, every epoch |
| Shard | Write large sequential files of token ids | Sequential write bandwidth | Leaving one file per sample, so training reads are metadata-bound |
| Retain | Tier and expire by access pattern | Cost per terabyte, and the retrieval cost of the cold tier | Keeping everything on the fast tier because deleting feels risky |
the processing cost, which decides where it runs
tokenizer throughput: roughly 1 MB of text per second per core
a 60 TB corpus: 60e12 / 1e6 = 60e6 core-seconds
= 60e6 / 86,400 = 694 core-days
as a one-off batch job on cheap CPU capacity: a few days on a modest fleet
repeated every epoch on the training cluster: 694 core-days of the CPUs that were supposed
to be feeding GPUs, per pass
sanity: the same work is unremarkable in one place and ruinous in the other, which is why
"tokenize offline" is the single most repeated piece of advice about data pipelines
Sharding is the decision that matters
one file per sample
a 60 TB text corpus at 4 KB per document is 15 billion objects
every training read is an open, a stat and a small read
the metadata layer, not the bandwidth, becomes the limit, and it fails as high system time
and low throughput with idle network
sharded into 256 MB files
60e12 / 256e6 = 234,000 shards
a reduction in object count of about 64,000 times
each read is a sequential stream, which is what both parallel filesystems and object
storage are good at
what a shard should contain
pre-tokenized ids rather than raw text, so no CPU work happens at read time
records concatenated with an index, so a worker can seek within a shard
enough samples that a worker reads a shard for a while rather than reopening constantly
sanity: 15 billion objects against 234,000 is the whole argument, and the loader problem that
looks like slow storage is usually this
Retention, and what is safe to expire
Three classes with different rules. Sharded token ids for the run in progress are hot and must be fast to read. Previous shard versions are warm, kept so a training run can be reproduced, and they expire on a stated schedule rather than never. Raw sources are cold and are kept because reprocessing needs them, which is the one class that cannot be deleted at all, since losing it means the dataset cannot be rebuilt.
The version discipline is what makes expiry safe. A run records the exact shard set and the processing configuration that produced it, so a shard version can be expired once no retained run depends on it. Without that record, nobody can say which shards a past run used and the answer to every expiry proposal is no.
What interviewers are listening for
Whether the candidate distinguishes the four bottlenecks. Saying storage is slow is where most answers stop; saying ingest is metadata-bound, processing is CPU-bound, sharding turns metadata into bandwidth, and the training read for text is a few megabytes a second is the answer that shows the pipeline was actually operated. The tokenize-once point is the second signal, and the third is having a retention policy at all, since a team with none is paying fast-tier prices for cold bytes and cannot say which data a past run used.
Key takeaways
- Four stages, four bottlenecks: ingest is metadata-bound, processing is CPU-bound, sharding is sequential write, retention is cost per terabyte.
- Tokenizing 60 TB of text is about 694 core-days, which is a one-off batch job and never a per-epoch cost on the training cluster.
- Sharding a 60 TB corpus into 256 MB files cuts the object count from about 15 billion to 234,000.
- Store pre-tokenized ids with an index inside each shard, so no CPU work happens at read time.
- Record the exact shard set per run, because that record is what makes expiring old versions safe.