Fleet and Reliability
Operate a shared accelerator fleet: derive placeable capacity, diagnose network and storage limits, interpret hardware evidence, and design admission, recovery and tenant policies. Finish with an acceptance runbook that states exactly what the tests establish.
A cluster can have free devices while a distributed job cannot fit, or report healthy nodes while useful throughput falls. Operating it requires connecting workload evidence to allocation, physical paths and recovery decisions.
The scheduler module starts with measurable capacity: what is assigned, what is active and what shape a queued job needs. It then adds coordinated admission and the trade between locality and waiting.
The fabric module follows traffic through compute, storage, management and hardware-control paths. Worked calculations cover shared bandwidth, small-read concurrency, checkpoint durability and rack limits under a specified failure.
The fleet module examines fault evidence and the checks that govern resource eligibility. You will quantify synchronized slowdown, false-positive quarantine and the relationship between repair turnaround and spare capacity.
The tenancy module connects those constraints to entitlement, reclaim and isolation commitments. The final project is a 32-node acceptance runbook: test definitions, approved thresholds, expected duration, recovery checks and the evidence required before a workload may use the cluster.
When you finish, you can
- Distinguish allocation, activity and job-specific placeability with explicit denominators
- Explain coordinated admission, readiness timeouts and queue-policy tradeoffs
- Compare topology-sensitive runtime with queue delay and completion objectives
- Map network roles and derive storage throughput, request-rate and concurrency limits
- Calculate rack capacity under normal and required failure-state power and cooling limits
- Interpret fault evidence and measure health-gate coverage, false positives and recovery cost
- Specify entitlement, borrowing, reclaim and resource-specific isolation for shared tenants
- Produce an acceptance runbook with reproducible criteria, recovery tests and release ownership
BEFORE YOU START · Foundations of AI Infrastructure, or the equivalent. Kubernetes familiarity helps and is not assumed; this course explains what is different about scheduling accelerators rather than teaching container orchestration from scratch.
Put the course into practice
Build a project after the linked lesson. Keep the scripts, measurements and written decision as evidence of your work. Hardware requirements and execution limits appear on each project.
- Premium · Service project
Observe a GPU service and distinguish missing signals from healthy hardware →
Device evidence, a local metrics exporter, dashboard configuration and an incident worksheet.
- Premium · Portfolio capstone
Defend an inference benchmark with quality gates and cost evidence →
A reproducible comparison, an eligible Pareto frontier and a written decision with a reversal condition.
After: Acceptance runbook
Syllabus
The scheduler
Distinguish allocated, active and placeable capacity; coordinate admission and failed startup; then compare topology-sensitive runtime against queue delay. Worked examples expose the costs of packing, reservations and placement constraints.
The fabric
Map network roles and shared bottlenecks, derive storage limits in bytes and operations, and calculate rack capacity under power, cooling and feed-failure constraints. Use evidence to separate plausible causes before changing the fleet.
The fleet
Interpret fault evidence without assuming a diagnosis, design health gates with measurable coverage and false-positive cost, and connect repair throughput to spare capacity and urgent escalation. Define containment, validation and ownership at every transition.
The tenancy
Define entitlement, borrowing and reclaim costs; specify isolation by resource; then write a cluster acceptance runbook with reproducible tests, explicit thresholds and accountable release decisions.