Question 1

Why do prefill and decode behave so differently, and why does that matter for the hardware you serve on?

Accepted Answer

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

Question 2

What is the KV cache, and why does it keep growing while a request is being served?

Accepted Answer

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

Question 3

What is the difference between static and continuous batching, and why did it change LLM serving?

Accepted Answer

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

Question 4

How does PagedAttention work, and what problem was it solving?

Accepted Answer

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Question 5

Define TTFT, TPOT and goodput, and tell me how you would measure each one in production.

Accepted Answer

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

Question 6

You have eight H100s and Llama 3.1 70B. How many concurrent users can you serve, and what changes the number?

Accepted Answer

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

Question 7

A long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?

Accepted Answer

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.

Question 8

When does speculative decoding speed up serving, and when does it break even or hurt?

Accepted Answer

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.

Question 9

Compare the KV cache footprint of multi-head, grouped-query and multi-head latent attention with numbers.

Accepted Answer

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.

Question 10

You need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?

Accepted Answer

Weight-only int4, fp8 weights and activations, or fp8 for the cache alone: each fixes a different bottleneck. Match the method to the resource you are short of, then run the evals that catch what perplexity misses.

Question 11

When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?

Accepted Answer

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

Question 12

Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?

Accepted Answer

The same 2,000 tokens are prefilled a thousand times an hour. Caching their KV by content turns that into one prefill and a table lookup, and a radix tree is what makes multi-turn and branching agents share it too.

Question 13

You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.

Accepted Answer

Sequential is fifty seconds; one batch is four. In between are the questions the interviewer is holding: how you pack ragged inputs, what a batch window costs, and where the memory stops you.

Question 14

Your p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.

Accepted Answer

A flat median with a broken tail means one in a hundred requests is hitting something the others do not. There are five usual suspects, and the order you rule them out in is the answer.

Question 15

vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?

Accepted Answer

Three engines, one hardware roofline, and the difference between them is which part of the roofline each one reaches first on your workload. The decision is a table, and the table has reversal conditions.

Question 16

How would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?

Accepted Answer

Three hundred fine-tunes as three hundred deployments would need three hundred GPUs. Batched adapters put them on one base, and the arithmetic says what each extra adapter costs in memory and in step time.

Question 17

Design an autoscaler for GPU inference replicas that reacts to load without thrashing.

Accepted Answer

A replica takes a minute to become useful and the traffic changes in seconds. Scale on the wrong signal and you buy GPUs you never use, or you drop requests while they boot. The design is three numbers and two timers.

Question 18

A new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?

Accepted Answer

141 gigabytes have to move from somewhere to eight GPUs, and every hop has a bandwidth. Add the CUDA init, the engine warm-up and the graph capture, and the minute is gone unless you design each step.

Question 19

What changes when you serve DeepSeek-V3 compared with a dense 70B model?

Accepted Answer

671 billion parameters that touch 37 billion per token: the memory bill is for the big number and the bandwidth bill is for the small one, until batch and routing collide. The cache is tiny, and that changes the whole fit.

Question 20

How do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?

Accepted Answer

Least-loaded routing sends a conversation's tenth turn to a replica that has never seen it, and the whole history prefills again. Affinity fixes that and creates hot spots. The router that does both is a scoring function with two terms.

Question 21

You stream tokens to clients over HTTP. What can go wrong, and how does the server handle each case?

Accepted Answer

A stream is a long-lived connection to a GPU slot. Slow readers, dropped connections, half-finished answers and duplicate retries each waste that slot in a different way, and each has a specific defense.

Question 22

You need to serve 128k-token contexts. What breaks first, and what do you change?

Accepted Answer

At 128k tokens one request's cache is 43 gigabytes and its prefill is measured in seconds. Capacity breaks first, then TTFT, then the scheduler. Each has a fix, and the numbers say which fix you need at which length.

Question 23

EAGLE, Medusa or a separate draft model: which speculative decoding method do you pick, and why?

Accepted Answer

All three guess tokens for the big model to check. They differ in where the guess comes from, how many candidates they verify per step, and how often the target agrees. Acceptance length per unit of draft cost is the number that decides.

Question 24

Draw the throughput versus latency curve for LLM serving and explain where you would operate on it.

Accepted Answer

Tokens per second climbs with batch and per-token latency climbs with it. The curve has a knee set by the hardware ridge, and the operating point is where your latency SLO cuts it.

Question 25

Design the fastest serving path for a 70B model with a 200 ms TTFT target at production scale.

Accepted Answer

Two hundred milliseconds is about four thousand prompt tokens of prefill on a full node, before any queueing. Every design choice is about not spending that budget twice.

Question 26

nvidia-smi shows 30 percent utilization on our serving fleet. Is that a problem, and what would you look at instead?

Accepted Answer

The number nvidia-smi calls utilization measures whether any kernel was running, not whether the chip was busy. A decode step that saturates HBM can read as thirty percent. The right dashboard has four other numbers on it.

Question 27

What changes when you serve a vision-language model instead of a text-only one?

Accepted Answer

An image is not one token; it is hundreds to thousands of them after a second model has run. That changes the prefill budget, the batch shape and where the first bottleneck appears.

Question 28

When does offloading the KV cache to CPU memory or NVMe beat recomputing it?

Accepted Answer

A paused conversation holds gigabytes of cache that its next turn will need in thirty seconds. Recompute costs prefill; reload costs bandwidth. The comparison is one division per tier, and the answer flips with prompt length.

Question 29

A customer sends the same prompt twice at temperature zero and gets different answers. Explain why, and what you can promise them.

Accepted Answer

Temperature zero removes the sampling randomness and leaves the floating-point kind. The batch your request lands in changes the reduction order, the logits move in the last bits, and a near-tie flips a token. The fix has a cost.

Question 30

Design an inference platform for 10,000 requests per second on a 70B model. Size it and name the SLOs.

Accepted Answer

Ten thousand requests a second is a fleet of hundreds of nodes, a router that has to know what every replica is caching, and an SLO pair that decides the batch on every one of them. Here is the sizing chain, node by node.