Together AI LLM Inference & Serving interview questions

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.

Weight-only int4, fp8 weights and activations, or fp8 for the cache alone: each fixes a different bottleneck. Match the method to the resource you are short of, then run the evals that catch what perplexity misses.

Three engines, one hardware roofline, and the difference between them is which part of the roofline each one reaches first on your workload. The decision is a table, and the table has reversal conditions.

671 billion parameters that touch 37 billion per token: the memory bill is for the big number and the bandwidth bill is for the small one, until batch and routing collide. The cache is tiny, and that changes the whole fit.

All three guess tokens for the big model to check. They differ in where the guess comes from, how many candidates they verify per step, and how often the target agrees. Acceptance length per unit of draft cost is the number that decides.

Tokens per second climbs with batch and per-token latency climbs with it. The curve has a knee set by the hardware ridge, and the operating point is where your latency SLO cuts it.

Ten thousand requests a second is a fleet of hundreds of nodes, a router that has to know what every replica is caching, and an SLO pair that decides the batch on every one of them. Here is the sizing chain, node by node.

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

The same 2,000 tokens are prefilled a thousand times an hour. Caching their KV by content turns that into one prefill and a table lookup, and a radix tree is what makes multi-turn and branching agents share it too.