mediumOpenAIAnthropic

A customer sends the same prompt twice at temperature zero and gets different answers. Explain why, and what you can promise them.

Greedy decoding chooses the highest-scoring token, but identical prompt text does not guarantee identical inputs or numerical execution. Check the effective request and deployment, investigate the first differing token, and define the reproducibility guarantee precisely.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🚀 Inference & Serving🔒 Premium

Tokens, Embeddings and LogitsTrace token IDs, embedding rows and next-token scores. Work out which tensors grow with a batch, when all-position logits waste memory, and why token probabilities do not establish factual confidence.

Advanced

📐 AI Systems Design🔒 Premium

Evaluation and Data Pipeline InfrastructureData pipelines prepare traceable training inputs; evaluation pipelines compare model behavior. Both need versioned inputs, reproducible execution and useful failure reporting.

Foundational

🚀 Inference & Serving

Prefill vs DecodeFollow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign an inference platform for 10,000 requests per second on a 70B model. Size it and name the SLOs.Next in this trackYour speech model runs faster than real time. Why do users still wait or hear gaps?Next in this trackWould request hedging improve an LLM endpoint whose p99 TTFT is too high?

FEDITOR'S NOTE

Verify the effective request and deployment before attributing variation to arithmetic. Locate the first differing token, explain a numerical mechanism, and scope any reproducibility promise to a tested execution environment.

DISCUSSION · 0

No comments yet — be the first to share your approach.