p99
AI infra interview questions tagged p99, across every topic.
3 questions · 1 unlocked for you
Concepts behind "p99"
The curriculum that explains the ideas these questions test.
Core
Designing for Latency SLOsA latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.📐 AI Systems DesignSign in
Advanced
Capacity and BackpressureA system that accepts more work than it can finish does not degrade gracefully; it degrades completely, because every request it queues makes every other request slower until all of them time out. The defence is backpressure: bounded queues at every stage, admission control that rejects early when the expected wait exceeds the budget, load shedding by priority, and clients that back off. For LLM serving the stages are the gateway, the router, the engine's queue and its KV pool. This page works the arithmetic of why unbounded queues kill p99, designs the bounds per stage, and the client contract that keeps it stable under overload.📐 AI Systems Design🔒 Premium