The model authors publish a new revision. How do you roll it out without a quality regression?

A model update is a deploy whose failures are invisible to every deployment metric. What has to be re-validated even for a point release, the canary that watches output shape rather than error rate, and the fingerprint that makes a customer report investigable.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieMost published serving numbers are not comparable to each other and not predictive of production, because they differ in the input distribution, the concurrency, whether the cache was warm, and which of several very different metrics is being reported. A benchmark that supports a decision has to fix all four, report a distribution rather than a mean, and be run against the traffic shape you actually serve. The single most useful discipline is to compute the bandwidth bound first, so you know what fraction of the possible you achieved.

Foundational

🧮 Open Weights & Serving Engines

Model Onboarding: From Hugging Face to ProductionA new open-weights model lands and someone asks how long until it is serving traffic. The answer depends on a sequence that is the same every time: read the card and the config, check engine support for the exact attention and quantization combination, size it, pull the weights, bring up one replica, validate correctness against the authors' own outputs, benchmark, then roll out behind a flag. The steps that surprise people are the download, which is hours for a trillion-parameter model, and the correctness check, which almost nobody does and which catches the wrong template.

Advanced

🧭 Ownership & Judgment🔒 Premium

Migrations and DeprecationsEvery infrastructure career contains a migration nobody wanted: the scheduler swap, the driver upgrade across a live fleet, the storage move while training runs are in flight, the deprecation of the launcher every team's scripts depend on. The behavioral round asks about one because it tests the skills that matter most and show least on a résumé: sequencing under risk, keeping a rollback real, moving people who have no reason to move, and knowing when to stop. This page gives the shape of a migration story that lands, two worked narratives from GPU fleet work, and the answers that sound like leadership and fail.

Advanced

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureBehind every model release is a pipeline that turns raw text into training shards and a harness that runs thousands of evaluation prompts against every checkpoint, and both are infrastructure problems with GPU-sized budgets. The data side is a batch system: dedup, filter, tokenize and shard petabytes with lineage. The eval side is a serving system in disguise: run a benchmark suite against a checkpoint in minutes, on shared GPUs, reproducibly, with results a researcher can trust. This page designs both, derives the compute and storage they need, and gives the reproducibility rules that separate a real harness from a script.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign a platform that serves twenty open-weights models of varying size to internal teams.Next in this trackAn SGLang deployment underperforms. Tune it against the project's own published targets.Next in this trackYou upgraded the serving engine and throughput fell 15 percent. Find the cause.

FEDITOR'S NOTE

Scored on re-running correctness validation even for a minor revision, on a canary watching quality signals, and on pinning and fingerprinting so a report can be tied to a version.

DISCUSSION · 0

No comments yet — be the first to share your approach.