Dataset Lifecycle: Ingest, Shard and Retain

TL;DR: A dataset lifecycle turns sources into a named, reproducible input and keeps its dependencies available for as long as readers need them. Record what arrived, version the transformations, publish verified shards through an immutable manifest, and pin that version in each run. Measure the bottleneck at each stage. Delete by retention policy and live references, not merely by age or by the arrival of a newer version.

From a folder to an identifiable dataset

A training run resumes successfully, but its loss changes sharply. The checkpoint names training/latest; overnight, another job replaced that folder's contents. The model state was restored, but the input changed. A dataset version gives the run an exact answer to “which examples, in which representation?”

A shard is a file or object containing multiple records. A manifest lists the shards belonging to one version, with identifiers, sizes, counts and checksums. A pin records that a run, retained checkpoint or other consumer still requires that version. These are application contracts; creating a bucket does not supply them automatically.

Ingest may be limited by request rate, bandwidth, decompression or source throttling. Processing may be CPU-, memory- or I/O-bound. Measure bytes, records, request latency, CPU time and queueing at each boundary before buying capacity. Parallel Filesystems vs Object Storage explains why the application's operations matter as much as total bytes.

Size the representation you actually store

Suppose an original planning case produces 120 billion token IDs, stored as unsigned 32-bit integers. Ignore indexes, metadata, padding and compression for this calculation; use decimal units.

payload = 120e9 tokens × 4 bytes/token = 480e9 bytes = 480 GB
at 256 MB/shard: ceil(480e9 / 256e6) = 1,875 shards
at 128 MB/shard: ceil(480e9 / 128e6) = 3,750 shards
sanity: halving shard size doubles the object count, not the token payload.

Raw text size cannot be substituted for token-ID size. Compression changes stored bytes, while indexes and packing add overhead. A shard size is a starting experiment: larger shards reduce request overhead for sequential consumption but may increase retries, wasted reads or worker imbalance. Preserve record boundaries and enough independently assignable work. Data Loading Pipelines for Training follows the next step from stored bytes to ready batches.

Precompute deterministic transformations when reuse justifies the build and storage cost. Keep intended per-epoch randomness explicit: online image augmentation, for example, may deliberately produce a new view each pass. Pre-tokenized input can still require reading, decompression, shuffling, packing and transfer. It does not remove all CPU work. Hugging Face's cache documentation describes transform fingerprints; a reusable local cache is useful, but it is not by itself a durable dataset publication or retention policy.

Publish a complete version, then pin it

Write shards under immutable identities. Verify all expected shards, then publish an immutable manifest referencing exactly those objects. Readers start from that manifest, not a listing of a folder that is still being filled. A friendly latest pointer may help discovery; resolve it once and save the immutable version in the run and checkpoint.

S3 documents atomic updates to one key, not transactions spanning multiple keys. A manifest-last protocol is an application design on top of that contract: use unique or version-pinned shard identities and prevent referenced objects from being overwritten or deleted. A completed manifest cannot protect mutable shards beneath it. Coordinate competing publishers if they update a shared discovery pointer.

The manifest identifies the dataset. It does not identify the next sample. Resuming also needs the sampler/shuffle state or equivalent deterministic input position associated with the saved model state; a read-ahead cursor can be ahead of consumed input.

Retain dependencies, not just version names

In this small example, v1 references shards A and B; v2 references A and C. A retained checkpoint pins v1. Assume v2 has been retired, its retention period has elapsed, and no reader or other policy needs it. Only C becomes a deletion candidate. A is shared and must survive.

This executable snapshot calculation illustrates shared references. It never deletes anything:

from math import ceil

assert ceil(120_000_000_000 * 4 / 256_000_000) == 1875
versions = {"v1": {"A", "B"}, "v2": {"A", "C"}}
objects = {"A", "B", "C"}

def deletion_candidates(objects, versions, retained):
    # Missing manifests or referenced objects must fail closed.
    required = set().union(*(versions[v] for v in retained))
    if not required <= objects:
        raise ValueError("retained dataset is incomplete")
    return objects - required

assert deletion_candidates(objects, versions, {"v1"}) == {"C"}
assert deletion_candidates(objects, versions, {"v1", "v2"}) == set()
print("v1 retained: candidate C; both retained: no candidates")

A real collector also protects active readers and in-progress publishers, coordinates pin acquisition against deletion, and rechecks candidates after a grace period. The snapshot above cannot resolve those races. Retain raw sources when the agreed reconstruction or audit requirement needs them and retention permits it. A replacement cache can expire earlier than its authoritative copy. Moving that copy to a cold tier is acceptable only if retrieval still meets the required recovery time.

Check your understanding

A new version is published. Can yesterday's shards expire? Only after checking every retained version, checkpoint and active reader. In the example, publishing v2 says nothing about whether A or B is safe to remove.

Would 128 MB shards halve the storage bill? No: the modeled payload remains 480 GB and object count doubles to 3,750. Requests, indexes, compression and useful read amplification determine the rest of the tradeoff.

What must a training restart name? An immutable dataset version plus the input state belonging to the checkpoint. Checkpoint I/O explains when that saved state actually becomes recoverable.

What to remember

In an interview, explain how a reader finds a complete version and how retention keeps it recoverable. The example stores 480 GB of token IDs in 1,875 shards; those numbers describe representation, not raw-source size. A version label alone protects nothing: the retained manifest, referenced shards and saved input state have to agree.