You have 60 terabytes of filtered text and need 15 trillion training tokens. Design the tokenization and sharding stage.

Plan a reproducible tokenization build: measure CPU demand, choose a safe token dtype, publish complete shards and distinguish token lookup from training resume.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Plan a reproducible tokenization build: measure CPU demand, choose a safe token dtype, publish complete shards and distinguish token lookup from training resume.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingTrace batches from storage to GPU readiness, budget workers and prefetch memory, and resume from the input position saved with training state. An executable example shows how a read-ahead cursor can silently skip samples.

Foundational

🔌 Networking & Storage

Dataset Lifecycle: Ingest, Shard and RetainFollow a dataset from source records to verified shards, a pinned version and safe retention. Calculate token storage, distinguish caches from authoritative data, and learn why deleting a version does not necessarily free its shards.

Advanced

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureData pipelines prepare traceable training inputs; evaluation pipelines compare model behavior. Both need versioned inputs, reproducible execution and useful failure reporting.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingConsistent hashing limits key movement when a shard set changes. It can preserve cache affinity, but it does not guarantee that cached data is present or that load is balanced.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign the checkpoint write path for a 405B model on 16,384 GPUs. What has to be true for the pause to stay under a minute?Next in this trackMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?Next in this trackWhich NCCL environment variables are worth knowing, what does each change, and which should you almost never set?

FEDITOR'S NOTE

Separate source bytes from tokens, qualify measured rates and preserve tokenizer identity across retries. Explain why token offsets alone cannot resume a shuffled loader.

DISCUSSION · 0

No comments yet — be the first to share your approach.