← Open-Weights Models & Serving Engines / 45

hardNewBasetenTogether AIFireworks AI

Your model now serves 6,568 tokens per second. What does a million tokens cost, and what do you hand the gateway and eval teams?

Turn a benchmark number into a bill that survives real utilization, then write the serving contract the gateway enforces and the evaluation the tuned configuration still owes.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Turn a benchmark number into a bill that survives real utilization, then write the serving contract the gateway enforces and the evaluation the tuned configuration still owes.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🧮 Napkin Math & Capacity🔒 Premium

Cost per Million TokensCost per million tokens connects fleet spending to useful output. Keep the time window, token definition and utilization assumptions consistent when comparing deployments.

🚀 Inference & Serving🔒 Premium

The Serving Optimization WaterfallTune an LLM deployment one change at a time against one fixed workload, in an order that follows the binding constraint. A measured run went from 789 to 6,568 output tokens per second, and the two biggest wins were not the knobs people usually reach for first.

🚀 Inference & Serving🔒 Premium

Reasoning Budgets and Answer LatencyReasoning tokens consume generation time and retained state before an answer becomes usable. Separate the first generated token from the first answer token, reserve output capacity, and evaluate reasoning limits against task success.

📐 AI Systems Design🔒 Premium

Offline Batch Inference vs Online ServingOnline serving promises each request a latency; a batch job promises a deadline for a whole set and a checked result for every item. That changes what to optimize, when an engine's startup cost dominates, and what delivery means.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYou need product copy for a 150-item catalog by tomorrow. Batch job or the online endpoint, and how do you deliver it?Next in this trackDeploy one open-weights model on Kubernetes with vLLM. What goes in the manifest, and what breaks first?Next in this trackYou serve a model with vLLM on GPUs. What changes when you move the same model to TPUs?

DISCUSSION · 0

No comments yet — be the first to share your approach.