scheduler

Translate request demand into prompt work, generated tokens and retained state, then size a measured serving configuration and its failure reserve.

Apply the project’s offline tuning guidance in context, derive the memory tradeoff, and use workload-specific acceptance criteria instead of universal queue and headroom targets.

Follow a four-GPU request from advertised capacity to concrete devices, and see why free GPUs across nodes cannot satisfy one pod.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.