Advanced

Ray on Kubernetes

Ray is a distributed compute framework that turns a cluster into one Python program: tasks and actors placed by Ray's own scheduler, with libraries for data loading, training, tuning and serving on top. KubeRay runs Ray clusters as Kubernetes resources. The combination is common in ML platforms because it gives researchers a single API for pipelines, distributed training and serving, and it is awkward in exactly one place: Ray has a scheduler and Kubernetes has a scheduler, and a platform engineer has to decide which one owns the GPUs. This page covers what Ray is good for, how KubeRay works, and where the two schedulers fight.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Kubernetes GPU Scheduling Gang Scheduling with Kueue and Volcano Containers, Images and GPU Cold Starts Data Loading Pipelines for Training Evaluation and Data Pipeline Infrastructure Inference Autoscaling and Cold Starts

LESSONS THAT TEACH THIS

Inference EngineeringChoose an engine by the layer it occupies, not by its throughput claim14 min→

PRACTICE THIS IN REAL QUESTIONS

Kubernetes, Slurm & GPU SchedulingWhat is Ray on Kubernetes good for, and where do its scheduler and the Kubernetes scheduler fight each other?→Kubernetes, Slurm & GPU SchedulingWe want to run LLM inference on Kubernetes. What does the deployment actually need beyond a Deployment with a GPU limit?→Napkin Math, Cost & CapacityTraffic peaks at three times the daily average. Capacity-plan the serving fleet.→LLM Inference & ServingDesign an autoscaler for GPU inference replicas that reacts to load without thrashing.→AI Infrastructure System DesignWalk me through an inference platform for a hosted LLM. What are the pieces, and what does each one do?→Open-Weights Models & Serving EnginesDesign a platform that serves twenty open-weights models of varying size to internal teams.→