← Open-Weights Models & Serving Engines / 46

mediumNewBasetenModalTogether AI

You need product copy for a 150-item catalog by tomorrow. Batch job or the online endpoint, and how do you deliver it?

A measured run shows where the time goes in a small batch job, then why every result parsed but none was deliverable. The answer turns on startup cost, co-tenant latency and a per-item contract.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

A measured run shows where the time goes in a small batch job, then why every result parsed but none was deliverable. The answer turns on startup cost, co-tenant latency and a per-item contract.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

📐 AI Systems Design🔒 Premium

Offline Batch Inference vs Online ServingOnline serving promises each request a latency; a batch job promises a deadline for a whole set and a checked result for every item. That changes what to optimize, when an engine's startup cost dominates, and what delivery means.

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching changes which requests run between model iterations. Learn how a waiting request joins, why an empty slot is not enough, and how to balance throughput against each user’s token latency.

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputRead a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

🧮 Open Weights & Serving Engines

Model Onboarding: From Hugging Face to ProductionTurn a model release into a reproducible service: pin the complete artifact, confirm the exact engine path, budget transfer and runtime memory, validate request and response behavior, then benchmark and canary against explicit acceptance criteria.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDeploy one open-weights model on Kubernetes with vLLM. What goes in the manifest, and what breaks first?Next in this trackYou serve a model with vLLM on GPUs. What changes when you move the same model to TPUs?Closest to this topicSet max-model-len and max-num-seqs for a chat product from first principles.

DISCUSSION · 0

No comments yet — be the first to share your approach.