scheduling

Each version fixed a different bottleneck: version 1 fixed memory traffic, version 2 fixed non-matmul work and parallelism, version 3 fixed the fact that softmax and matmul were waiting for each other on Hopper. The arithmetic that shows why a softmax costing one percent of the FLOPs can cost sixty percent of the time.

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.

A tail that wide is queueing or prompt-length variance, and the two are distinguishable in one measurement. Four causes ranked by how often they are it, the decomposition that attributes the wait, and the fix that is usually a flag rather than hardware.

Same job, same GPU count, half the speed: the slow one was split 4 and 4 across two nodes and its tensor-parallel all-reduces run at fabric speed instead of NVLink speed. The bandwidth arithmetic, the per-parallelism placement rule, and the scheduler policy that enforces it.

Device plugins count GPUs and know nothing else; Dynamic Resource Allocation lets a pod ask for devices by attribute and share them by claim. What the two models can and cannot express, what DRA changes for MIG, NVLink domains and multi-node gangs, and a migration stance for a fleet that runs both.

The expensive thing is a base model in GPU memory, not the customer's job. Keep bases resident, pack LoRA jobs onto them, isolate tenants at the process and device level, checkpoint adapters in seconds so jobs run on cheap capacity, and attribute cost per tenant. The arithmetic behind each rule.

A heap orders the queue and the interesting logic is elsewhere: which running jobs may be evicted, how many, and what happens to them. The victim-selection rule that avoids evicting more than necessary, the equal-priority case that must not preempt, and the starvation the design creates if nothing ages.

Three one-line policy differences produce different fragmentation, and the metric that separates them is not utilization. The measured outcome on the same job sequence, why the largest free block is what matters on a training cluster, and the case where the intuitive policy is exactly wrong.

Admission, a step that grows every sequence, and preemption under memory pressure. The loop is short and it has a failure that only appears when the cache is tight: in a measured run it preempted seventeen times and completed nothing, because every preempted sequence restarted from zero.

Fair sharing of one resource is obvious and fair sharing of three is not, because a GPU-heavy job and a memory-heavy job are not comparable. The rule that makes them comparable, the greedy loop that implements it, and the tie-breaking bug that gave one user everything.