hardNewOpenAIGoogleFireworks

What changes when you serve a vision-language model instead of a text-only one?

An image is not one token; it is hundreds to thousands of them after a second model has run. That changes the prefill budget, the batch shape and where the first bottleneck appears.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

An image is not one token; it is hundreds to thousands of them after a second model has run. That changes the prefill budget, the batch shape and where the first bottleneck appears.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🚀 Inference & Serving

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Core

🚀 Inference & ServingSign in

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Advanced

🚀 Inference & Serving🔒 Premium

Multi-LoRA ServingA LoRA adapter is a few hundred megabytes of low-rank matrices that turn a base model into a fine-tuned variant, and multi-LoRA serving runs hundreds of them on one copy of the base weights by keeping the adapters in memory and applying the right one per request inside the batch. It is how a platform serves a thousand customers' fine-tunes without a thousand deployments. The costs are an extra small matmul per layer, adapter memory and loading, and a scheduler that has to batch across adapters without starving any of them.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhen does offloading the KV cache to CPU memory or NVMe beat recomputing it?Next in this trackA customer sends the same prompt twice at temperature zero and gets different answers. Explain why, and what you can promise them.Next in this trackDesign an inference platform for 10,000 requests per second on a 70B model. Size it and name the SLOs.

FEDITOR'S NOTE

Scored on quantifying image tokens and the encoder's cost, on placing the encoder in the pipeline (separate stage versus in-engine), and on batching heterogeneity between image-heavy and text-only requests.

DISCUSSION · 0

No comments yet — be the first to share your approach.