mediumNewOpenAIAnthropic

A customer sends the same prompt twice at temperature zero and gets different answers. Explain why, and what you can promise them.

Temperature zero removes the sampling randomness and leaves the floating-point kind. The batch your request lands in changes the reduction order, the logits move in the last bits, and a near-tie flips a token. The fix has a cost.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureBehind every model release is a pipeline that turns raw text into training shards and a harness that runs thousands of evaluation prompts against every checkpoint, and both are infrastructure problems with GPU-sized budgets. The data side is a batch system: dedup, filter, tokenize and shard petabytes with lineage. The eval side is a serving system in disguise: run a benchmark suite against a checkpoint in minutes, on shared GPUs, reproducibly, with results a researcher can trust. This page designs both, derives the compute and storage they need, and gives the reproducibility rules that separate a real harness from a script.

Foundational

🚀 Inference & Serving

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Core

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign an inference platform for 10,000 requests per second on a 70B model. Size it and name the SLOs.Closest to this topicYou have 60 terabytes of filtered text and need 15 trillion training tokens. Design the tokenization and sharding stage.Closest to this topicDesign the eval pipeline for a frontier model: thousands of evals per checkpoint, sharded inference, caching, reproducible results.

FEDITOR'S NOTE

Scored on naming the actual source (batch-dependent kernel reduction order, not sampling), on knowing that a flipped token at step 40 changes everything after it, and on giving a precise promise with its cost.

DISCUSSION · 0

No comments yet — be the first to share your approach.