TL;DR: Choose storage from access semantics and measured workload, not a universal price or latency table. An object store can hold versioned durable data while a filesystem or local cache serves a working set. A second tier earns its cost only when it improves useful access enough to justify staging and consistency work.
How to approach it
Describe read sizes, concurrency, mutation and required filesystem behavior. Calculate bytes and requests separately. Then size any hot tier from concurrent working sets and explain how a job knows the staged data is the right version and complete.
A strong answer
Lustre is a parallel filesystem: files are striped across many storage servers, and clients see one shared directory tree. An object store such as S3 stores whole objects behind an HTTP API instead. A fast sequential-read benchmark tells you little about a loader opening millions of tiny files. It also tells you little about an object store with enough concurrent readers to hide request latency. Parallel Filesystems vs Object Storage frames the choices; the access pattern determines which costs matter.
A parallel filesystem can provide filesystem semantics and shared access with distributed data and metadata services. An object store exposes its own object APIs and semantics. Do not assume every object service lacks all partial-update or rename-like features, or that every filesystem has the same durability. Name the required operations and verify the chosen service's contract.
A useful two-tier example is FSx for Lustre integration with S3. It is one supported product arrangement, not evidence that all clusters need those products or two tiers. Direct object streaming, local NVMe staging and a shared filesystem can each be appropriate.
Request overhead matters even when total bytes fit. Under a synthetic serialized-stream model with 50 ms request latency and 1 GB/s payload rate:
100 MB shard: 0.05 + 0.10 = 0.15 s; latency fraction = 1/3
1 GB shard: 0.05 + 1.00 = 1.05 s; latency fraction ≈ 4.76%
sanity: larger shards amortize latency, but concurrency can hide it too.
Larger shards are not free. They affect shuffle quality, parallel assignment, retries, cache eviction and how much data a failed read must repeat. Choose a format with indexing or a streaming plan that preserves the intended sampling distribution. Data Loading Pipelines for Training connects those choices to batch readiness.
For metadata, suppose an access pass opens 300 million individual files. Grouping 10,000 documents per shard reduces the open count to 30,000. That is a 10,000× request-count reduction, not automatically a 10,000× epoch speedup. Metadata can overlap payload work; caching and multiple services change observed rates. Measure open/lookup latency and queueing as well as bytes per second.
Size the hot tier explicitly. If three concurrent jobs need 8, 12 and 20 TB of distinct data, their union is 40 TB. With 25% capacity allowance it becomes 50 TB. Shared shards can reduce the union; independent caches can replicate it. Reserve room for partial downloads and eviction behavior rather than filling every byte with a nominal working set.
The dataset representation matters too. Fifteen trillion token IDs use 60 TB at four bytes each before indexes and metadata. Two-byte IDs would be 30 TB but can represent only 65,536 distinct unsigned values. A tokenizer with a larger vocabulary cannot simply be stored as uint16 without an additional encoding scheme. Preserve exact IDs and document boundaries.
For a hypothetical 30 TB staging job at a sustained 5 GB/s, payload time is 6,000 seconds, or 100 minutes. Either stage before allocation, stream ahead of consumption, or include that delay in job cost. Mark readiness only after verifying the manifest and required content. A partially downloaded file under the expected name must not be mistaken for a complete shard.
Choose a hot tier when repeat use, latency or semantics justify it. Reverse the plan if direct streaming meets readiness targets and the cache adds little value. Compare actual storage, requests, transfer charges, support and idle-GPU cost using current quotes; no generic per-TB price determines the answer.
What interviewers probe next
- "Can the hot tier be the only copy?" Only if its durability and retention meet the requirement; do not assume a cache does.
- "What if two dataset versions share names?" Key by immutable version or content identity and verify the manifest.
- "What should I monitor?" Ready-shard latency, cache hit rate, metadata/request tails and useful training throughput.
Common mistakes
- Comparing bandwidth while ignoring request counts and semantics.
- Storing large-vocabulary token IDs in an insufficient integer type.
- Treating staged filenames as proof of complete, current data.
Key takeaways
- The toy 1 GB shard spends about 4.76% of serialized read time on latency.
- Three distinct working sets totaling 40 TB require 50 TB with the stated allowance.
- Use versioned readiness and measured access costs to justify a second tier.