Inference performance
Make a model serve more users per GPU without breaking the latency you promised.
You want to own a serving fleet: throughput, time to first token, cost per token, and the scheduler decisions behind all three.
Serving mechanics and capacity arithmetic, a design round on batching and the KV cache, and a debugging round where p99 moved and p50 did not.
The course sequence
In this order. Each assumes the one before it.
The question tracks to drill
91 questions across 3 tracks, in the order this loop weights them. Design an inference platform, a batching system, a training cluster, with TTFT and TPOT numbers on the board.
Design an inference platform at 10k requests per second, a 10,000-GPU training cluster, a job scheduler with preemption and checkpointing, a serverless GPU runtime with sub-second cold starts, a multi-tenant fine-tuning service, an eval pipeline. The whiteboard round at OpenAI, Anthropic, Baseten and Together.
31 questionsPrefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend.
30 questionsMemory footprints, 6ND, arithmetic intensity and the ridge point, bandwidth-bound decode, communication volume, GPU counts and time to train, cost per million tokens, TCO and buy versus rent. The estimation round almost every AI infra loop includes, with every assumption stated.
30 questionsWho hires for this
Grouped by the kind of employer, because archetype predicts the loop better than the brand does.