What Does an AI Infrastructure Engineer Do All Day?

Kernel and performance engineers spend their days in profilers. A typical unit of work is a model step that runs at 38% MFU when the roofline says 55% is available: Nsight Systems to find which kernels dominate, Nsight Compute to see why the attention kernel is memory-bound at a shape it should not be, a fused kernel in Triton or CUTLASS, a benchmark harness that proves the gain across shapes, and a review from a colleague who will ask about numerical drift in bf16. The NVIDIA, Anthropic and OpenAI postings describe exactly this loop: find the gap between achieved and peak, close it, prove it. The mechanics are in CUDA and kernels.

Training infrastructure engineers own a run. The morning is the dashboard: step time, MFU, loss, the per-rank timing view that shows a straggler, the checkpoint cadence and the last successful write. The afternoon is a change: a new parallelism configuration for the next model size, a checkpoint writer that goes asynchronous, an elastic restart path, a NCCL tuning for a new fabric. The Microsoft AI and AWS Neuron postings describe implementing new distributed parallelism schemes and profiling at thousands of GPUs; the distributed training track covers the substance.

Inference and serving engineers live between an SLO and a bill. A representative week: a new model version raised p95 TPOT by 12 ms and the router's prefix affinity needs re-tuning; a customer with 100k-token prompts is filling KV pools and needs its own lane; the quantized variant lost a point on an eval and the decision is whether the 40% cost reduction is worth it; the autoscaler is scaling on the wrong signal. Anthropic's Inference Systems, DeepMind's Model Inference and the inference-provider postings (Together AI, Fireworks, Baseten) describe holding a fleet to throughput, latency and correctness bars. The topics are inference and serving.

Fleet and platform engineers run the cluster as a product. The queue of work: a driver rollout across 1,200 nodes without stopping a frontier run, a new delivery of 40 nodes going through burn-in, a team that bypassed the scheduler and fragmented a pod, the quota policy review, an NVLink error rate that is climbing on one rack, and the Kubernetes operator that provisions nodes. OpenAI's Fleet Infrastructure and HPC postings and CoreWeave's fleet validation posting describe the shape. The topics are scheduling and reliability.

ML platform engineers at product companies build the internal service researchers and product teams use to train, deploy and monitor: Uber's Michelangelo, Netflix's ML Platform, Databricks' Mosaic AI and Snowflake's Cortex postings describe feature infrastructure, model serving APIs, Ray and Spark, and multi-tenant platform design. Their week is closer to a backend engineer's, with GPUs as the expensive resource being scheduled.

Monday: the straggler alert fired twice over the weekend and auto-drained both nodes; the morning is reading the counters on the drained nodes (one NVLink cable, one thermal), filing the RMA, and adjusting the alert threshold that let the cable go undetected for two days. The afternoon is the capacity review with three research teams, where the arithmetic of a 1,024-GPU request against a pod's borrowable capacity gets done on a whiteboard.

Tuesday and Wednesday: the driver upgrade. The quarantine pool moved last week, the eval pool moves today, and the first training pod moves at a job boundary tomorrow night; the rollback image is on every node and was exercised on Friday. Wednesday's canary catches a CUDA compatibility issue in one team's container, and the image matrix gets a new rule.

Thursday: a 512-GPU job is 20% slower than last week and nothing failed. Per-rank timing finds the node, the link counters find the cable, the swap takes an hour, the job restarts from its checkpoint. The postmortem is short and its action item is an automation. Friday is the burn-in report on the new delivery (two of forty nodes failed, on the bench rather than in a job), the on-call handoff, and the fleet-health note that researchers read.

The on-call reality is specific to this fleet: pages are hardware (XIDs, link errors, thermal) more often than software, the reflex is drain and restart rather than debug in place, and the good weeks are the ones where the automation handled it and the on-call read about it in the morning. Meta's Llama 3 report is the reference point: 419 unexpected interruptions in 54 days on 16,384 GPUs, three of which needed a human. That ratio is the goal of every fleet team, and it is what the incident-response material on this site teaches.

It is not model research; you will read the papers that change the infrastructure (MoE routing, attention variants, low-precision training) and you will not be asked to design an architecture. It is not generic cloud infrastructure; a Kubernetes expert without GPU, fabric and performance knowledge is at the start of the learning curve, not the end. It is not a dashboards job; the postings say automation over dashboards in as many words, and the interviews test it. And it is not a role where the on-call is quiet: a fleet of this size has something wrong with it at every moment, and the craft is deciding which wrong things matter.

Across the tracks: Linux, Kubernetes (operators, controllers, the GPU operator, Kueue or Volcano), Slurm, InfiniBand and RoCE tooling, NCCL and its tests, DCGM, Nsight Systems and Compute, PyTorch (DDP, FSDP), JAX on TPU teams, Megatron and NeMo, vLLM, SGLang or TensorRT-LLM, Ray, Lustre or GPFS, object storage, Terraform and Ansible, Prometheus and Grafana, and one of Python, Go, C++ or Rust as the daily language. The NVIDIA new-grad posting lists most of this in one line, which tells you where the floor is. The skills guide maps each to what the loops test.