TL;DR: An evaluation run needs immutable case identities, a pinned execution manifest, retry-safe result commits and versioned scoring. Size inference from measured throughput on the actual case mix. Reuse only outputs whose full dependencies match, preserve independent sampling replicates, and prove completion by expected case IDs. Report uncertainty alongside score changes.
How to approach it
Clarify suites, prompt/output lengths, sampling, tools, checkpoint cadence, deadline and release criteria. Separate generation from grading. Define the run manifest and case identity before the queue. Then walk through a worker retry, a cache lookup and a comparison between checkpoints.
A strong answer
A report says four of four examples finished. The result table contains A, B, B and D. Counting rows declares victory while case C has never run. This is the kind of infrastructure error that turns an apparently precise model score into an unreliable one.
A run manifest records the exact ingredients of the evaluation: model artifact hash, tokenizer, chat template, task data version and generation settings. It also fixes the engine and precision configuration, tools or simulators, grader and metric definitions. Save random seeds and the execution details needed to explain a later difference. Create a case ID from the task/sample identity and a separate replicate ID for each intended stochastic trial. Identical prompts can be distinct experiments; deduplicating them blindly changes the evaluation.
Workers lease cases (take a time-limited claim on each one, which expires if the worker dies), run inference and conditionally commit one accepted result per run/case/replicate identity. Lease expiry can cause duplicate execution; the commit boundary prevents two accepted answers from the retry. Fence stale attempts and keep diagnostic attempt records separate. Completion compares the expected identity set with accepted terminal outcomes. Failed and missing cases must remain visible; a partial report cannot silently drop them from the denominator.
For a hypothetical ten-million-prompt suite averaging 1,500 input and 500 output tokens, generation consumes 15 billion input and five billion output tokens. Suppose a representative replay measures 20 complete prompts/s per worker with that joint length mix, and 32 workers sustain the same rate together.
Hypothetical measured joint throughput = 20 complete prompts/s per worker
32-worker pool = 20 × 32 = 640 complete prompts/s
Generation time = 10,000,000 / 640 = 15,625 seconds ≈ 4.34 hours
Sanity: 640 × 15,625 = ten million prompts, before retries and other overhead.
Grading, startup and stragglers add time. Independent prefill and decode peak rates cannot be added to predict shared-worker capacity. Benchmark the largest useful batch, not merely the largest batch that fits memory.
Keep caches at explicit dependency boundaries. A response cache must include checkpoint identity, formatted input and generation dependencies, so a new checkpoint normally misses. A grader cache can reuse a result only when the generated answer and every grading input/version match. A prompt-processing cache is a different object. The EleutherAI evaluation harness interface exposes separate response/request caching, sample logging, seeds and task configuration; pin the deployed version and inspect what its cache keys cover.
Scoring may use exact checks, execution tests, model judges or human review. Run untrusted generated code and tool calls in a sandbox with bounded resources and scoped credentials. Version a model judge just as carefully as the model being evaluated. Keep enough sample-level evidence to reproduce a score while applying the dataset's privacy and retention rules. Top-k token probabilities do not provide full perplexity unless the needed target-token probability is actually available.
For an illustrative independent Bernoulli task with 5,000 cases and 80% accuracy, the estimated standard error is sqrt(0.8 × 0.2 / 5000) ≈ 0.00566, or 0.566 percentage points. A simple normal 95% interval is roughly ±1.11 points. This single-score interval does not decide whether a paired 0.5-point change is meaningful: compare the same cases and inspect discordant outcomes or a suitable paired interval. Related questions and multi-turn trajectories may require cluster-level resampling. Repeated model selection against one benchmark can also overfit the release criterion.
I would retain immutable per-case results and versioned graders, with a small smoke suite before the expensive run. I would accept cache reuse only after testing dependency invalidation with changed templates, checkpoints and grader versions. Track missing identities, duplicate attempts, failure rate by task, p99 case duration, grader backlog and score changes by meaningful slice. Evaluation and Data Pipeline Infrastructure connects the stages; Capacity and Backpressure applies when graders fall behind generation. A faster aggregate report is not useful if tool failures or long prompts disappear from it.
What interviewers probe next
- How do you compare stochastic checkpoints? Preserve intended replicate identities and use a comparison method suited to paired cases and correlated trajectories. Do not collapse repeats because their prompts match.
- Can failed tool cases be excluded? Only under a declared reporting rule, with counts and reasons visible. Silent exclusion changes both the denominator and the claim.
Common mistakes
- Declaring completion from row counts while retries hide missing identities.
- Reusing responses across checkpoint changes or applying a single-score error bar to a paired change without analysis.
Key takeaways
- Four rows A/B/B/D do not complete expected cases A/B/C/D.
- The toy 32-worker pool needs 4.34 hours at a measured joint 20 prompts/s per worker, before overhead.
- New checkpoint identity invalidates response reuse; independent sampling replicates retain distinct IDs.
- An 80% score on 5,000 independent cases has about 0.566-point standard error; paired change analysis is a separate calculation.