Advanced

Ray on Kubernetes

Ray schedules Python tasks and actors across workers; KubeRay manages those workers on Kubernetes. The two scheduling layers need compatible resource and placement rules.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Kubernetes GPU Scheduling Gang Scheduling with Kueue and Volcano Containers, Images and GPU Cold Starts Data Loading Pipelines for Training Evaluation and Data Pipeline Infrastructure Inference Autoscaling and Cold Starts

LESSONS THAT TEACH THIS

Inference EngineeringChoose an engine against explicit responsibilities14 min→

PRACTICE THIS IN REAL QUESTIONS

Kubernetes, Slurm & GPU SchedulingWhat is Ray on Kubernetes good for, and where do its scheduler and the Kubernetes scheduler fight each other?→Kubernetes, Slurm & GPU SchedulingWe want to run LLM inference on Kubernetes. What does the deployment actually need beyond a Deployment with a GPU limit?→Napkin Math, Cost & CapacityTraffic peaks at three times the daily average. Capacity-plan the serving fleet.→LLM Inference & ServingDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.→AI Infrastructure System DesignWalk me through an inference platform for a hosted LLM. What are the pieces, and what does each one do?→Open-Weights Models & Serving EnginesDesign a platform that serves twenty open-weights models of varying size to internal teams.→