You have 60 terabytes of filtered text and need 15 trillion training tokens. Design the tokenization and sharding stage.

A tokenizer moves about a megabyte of text per second per core, which makes this a seven-hundred-core-day batch job rather than something to run during training. The throughput arithmetic per stage, the shard format the loader needs, and the determinism requirements that let you resume without corrupting a run.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingThe dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.

Foundational

🔌 Networking & Storage

Dataset Lifecycle: Ingest, Shard and RetainA training dataset is not a file, it is a pipeline with four stages and a retention policy, and each stage has a different bottleneck. Ingest is metadata-bound rather than bandwidth-bound. Tokenization is CPU work that should happen once offline rather than every epoch. Sharding decides whether the training read is a stream or a storm of small files. And retention decides how much of the bill is paid for bytes nobody reads.

Advanced

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureBehind every model release is a pipeline that turns raw text into training shards and a harness that runs thousands of evaluation prompts against every checkpoint, and both are infrastructure problems with GPU-sized budgets. The data side is a batch system: dedup, filter, tokenize and shard petabytes with lineage. The eval side is a serving system in disguise: run a benchmark suite against a checkpoint in minutes, on shared GPUs, reproducibly, with results a researcher can trust. This page designs both, derives the compute and storage they need, and gives the reproducibility rules that separate a real harness from a script.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingSplitting work across N servers with a modulo of N moves almost everything when N changes, which for a cache means throwing away almost all of it. Consistent hashing places servers and keys on a ring so adding or removing one moves only its share, and virtual nodes fix the imbalance a small ring otherwise has. In LLM serving the same structure routes requests by prompt prefix so a conversation reaches the replica already holding its cache.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?Next in this trackMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?Next in this trackWhich NCCL environment variables are worth knowing, what does each change, and which should you almost never set?

FEDITOR'S NOTE

Scored on the core-days arithmetic that sets the cluster size, on the output format decisions (shard size, dtype, manifest) tied to the loader's needs, and on determinism and resumability as first-class requirements.

DISCUSSION · 0

No comments yet — be the first to share your approach.