Advanced

Offline Batch Inference vs Online Serving

Online serving promises each request a latency; a batch job promises a deadline for a whole set and a checked result for every item. That changes what to optimize, when an engine's startup cost dominates, and what delivery means.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Continuous Batching Designing for Latency SLOs Capacity and Backpressure Cost per Million Tokens Containers, Images and GPU Cold Starts GPU Job Scheduler Design

LESSONS THAT TEACH THIS

Inference EngineeringA batch job is judged at its last item, not its average throughput22 min→Inference EngineeringCapstone lab: deploy one model, tune it, run a batch and hand it off360 min→

PRACTICE THIS IN REAL QUESTIONS

Open-Weights Models & Serving EnginesYou need product copy for a 150-item catalog by tomorrow. Batch job or the online endpoint, and how do you deliver it?→AI Infrastructure System DesignDesign an asynchronous batch inference API for a billion records with a completion deadline→Open-Weights Models & Serving EnginesSet max-model-len and max-num-seqs for a chat product from first principles.→Kubernetes, Slurm & GPU SchedulingOne fleet: training that wants every idle GPU, and inference with a p99 SLO. Separate pools, or one pool with preemption? Show the numbers.→Open-Weights Models & Serving EnginesDesign the benchmark you would run before putting a new open-weights model into production.→Napkin Math, Cost & CapacitySize an inference fleet for a 70B model serving 1,000 concurrent users→