Design the eval pipeline for a frontier model: thousands of evals per checkpoint, sharded inference, caching, reproducible results.

TL;DR: An eval run is a batch inference job: expand every eval into (prompt, sampling params) records, dedupe and cache by a content hash that includes the checkpoint, shard the misses across a pool of engine replicas running at the largest batch the KV pool allows, score the outputs with graders that are themselves cached, and store results keyed by (checkpoint, eval version, engine version, seed). At 2,000 evals × 5,000 prompts = 10 M prompts of 1,500 in and 500 out, a 70B checkpoint in fp8 takes about 130 node-hours, so 32 nodes turn a checkpoint around in about 4 hours. Reproducibility means pinning everything in the key and storing logprobs, not hoping temperature 0 is deterministic.

How to approach it

Ask how many evals, how many prompts each, how often a checkpoint arrives, how fast the number is needed, and whether the model is served through the production engine or a dedicated pool. Say the pipeline is batch inference plus a cache plus a scoring stage, and that turnaround is set by prompt volume over pool throughput. Draw the stages, size the run, then take caching and reproducibility as the deep dives.

A strong answer

A typical situation: a training run saves a checkpoint every 12 hours, the team wants the full suite (2,000 evals, average 5,000 prompts, some multi-turn, some with LLM-as-judge grading) on every checkpoint, and the number has to be trusted enough to decide whether to roll back a data change. Evaluation and Data Pipeline Infrastructure is the reference shape.

Sizing the run.

volume per checkpoint
  2,000 evals × 5,000 prompts = 10 M prompts; 1,500 in, 500 out
  prefill 15 G tokens; decode 5 G tokens
supply per 8 × H100 node, 70B fp8, batch as large as the KV pool allows since no user is reading
  prefill at MFU 0.4: 45,000 tok/s
  decode at batch 512, 2k contexts: step = (70.6 + 512 × 0.33) GB ÷ 26.8 TB/s = 8.9 ms → 57,000 tok/s roofline;
    take 40,000 tok/s
node-seconds: 15e9 ÷ 45,000 ≈ 333,000; 5e9 ÷ 40,000 = 125,000; total ≈ 458,000 ≈ 127 node-hours
on 32 nodes: 4 hours per checkpoint; on 8 nodes: 16 hours, longer than the checkpoint interval
cost: 127 × 8 × $2.50 ≈ $2,500 per checkpoint, twice a day
with a 30% cache hit rate across checkpoints (shared prefixes and unchanged deterministic evals are not
  hits, since the checkpoint is in the key; hits come from grader calls and repeated prompts within a run):
  the grader stage, which is another 10 M judge calls, is where the cache pays
sanity: the eval pool needs to be about a quarter of the size of the production pool it validates,
        which is why labs run it on a dedicated slice rather than borrowing serving capacity

Sharding. Each shard is a list of record IDs, not prompts, sorted by prompt length so a batch has similar prefill cost and the KV pool is used evenly. A coordinator hands shards to replicas with a lease; a replica that dies loses its lease and the shard is reassigned, with completed records already in the content-addressed store so nothing is redone. Multi-turn evals run as a chain: turn n's output is the input to turn n + 1, so those records are scheduled as a dependency graph, and their prefix is a cache hit on the same replica.

The cache key. The key is a hash over everything that can change an output: checkpoint hash, the prompt bytes, the sampling parameters, the engine version and its kernel configuration, the tensor-parallel layout, and the seed. Leaving the engine version out means an engine upgrade silently changes numbers under an unchanged key; leaving the layout out means TP4 and TP8 results are mixed. Grader calls use the same rule with the judge model's own checkpoint in the key, and that is where reuse across checkpoints happens: a judge scoring an identical output is a hit.

Reproducibility. Temperature 0 is not deterministic across batch compositions: floating-point reductions in attention and GEMM kernels differ by batch size and by which sequences share a step, so the same prompt can produce a different token at a near-tie. The design therefore records logprobs of the top-k at every position, pins the engine build and the layout, fixes the seed for sampled evals, and reports confidence intervals from the number of prompts rather than a single accuracy. A regression is reported with the per-prompt diff between two checkpoints, not just two numbers, so a reviewer can see whether 40 prompts flipped or 4,000.

The trade-off to commit to: a dedicated eval pool at the largest batch, rather than routing eval traffic through the production serving pool. It costs a fleet slice that sits idle between checkpoints and buys a turnaround the training team can plan around, with no TPOT SLO limiting the batch. The reversal condition: a small team whose checkpoints arrive weekly, where the pool would idle 95% of the time; there, run evals through production at low priority and accept the longer turnaround. Evaluation and Data Pipeline Infrastructure is where this harness lives, and Capacity Planning and Utilization is how its ten million prompts get scheduled.

Failure modes to name: a grader model update that changes every score (version it, and re-grade a fixed reference set to detect drift); an eval whose prompts leaked into training data (hash-match against the training corpus at expansion time); a replica running a different engine build than the key says (the agent reports its build, and the coordinator refuses the mismatch); a results table that mixes seeds; a partial run reported as complete (completion is by record count against the expansion, not by shard count).

What interviewers probe next

• "The checkpoint interval is 12 hours and the run takes 16; what do you do?" Tier the suite: a 200-eval smoke set on every checkpoint in under an hour, the full suite every second checkpoint, and the pool sized so the full suite fits inside two intervals.
• "Why store logprobs and not just the text?" A near-tie flip explains a regression in one look, and logprob-based metrics (perplexity, calibration) come free.
• "How do you know the number is real and not noise?" The interval from the prompt count: 5,000 prompts at 80% accuracy has a standard error of about 0.6 points, so a 0.5-point move is noise and the per-prompt diff says which.

Common mistakes

• Running evals through the chat endpoint at chat batch sizes and wondering why a checkpoint takes two days.
• A cache key without the engine version or the layout.
• Trusting temperature 0 as deterministic.
• Reporting a single accuracy with no interval and no per-prompt diff.

Key takeaways

• 10 M prompts per checkpoint on a 70B is about 130 node-hours; 32 nodes give a 4-hour turnaround.
• Cache key = hash(checkpoint, prompt, params, engine build, layout, seed); the grader stage is where hits live.
• Record logprobs, pin the engine and layout, fix seeds, report intervals and per-prompt diffs.
• A dedicated pool at the largest batch, tiered into a smoke set and a full suite.