autoscaling

Forecast demand across startup delay, track replicas already being created, and drain slowly enough to avoid repeating the same scale decision.

Translate request demand into prompt work, generated tokens and retained state, then size a measured serving configuration and its failure reserve.

Design a shared catalogue with explicit service tiers, reproducible deployments and bounded cold starts. Residency depends on burst behavior and resource reuse; placement depends on GPU groups and interference as well as memory.

Size for a tested peak and a stated failure, then price an explicit daily scaling schedule. Keep request-slot occupancy separate from GPU utilization.

Trace Kubernetes pod placement and Ray bundle reservation as separate layers, including logical GPU limits and autoscaling delay.

Make readiness mean that the engine can serve, scale from measured demand and ready capacity, and drain streams before termination. Kubernetes can implement these controls; an operator packages and reconciles them.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

A 60-second demand ramp meets a 70-second startup delay. Compare the rate and ready-capacity curves, then design a controller that tracks pending launches without a blind cooldown.