THE AI infra CURRICULUM
Understand the concepts before you drill the questions
A structured path through the ideas AI infrastructure and Applied AI loops actually test. Each concept gives you the intuition, a worked example, and the trade-off interviewers probe, then links straight to the real questions where it shows up. Read it like a curriculum, or jump to whatever you are weakest on.
150 concepts across 13 tracks · foundational concepts are free · see them as a map
Begin the curriculum →01
🧩 GPU & Accelerator Architecture
How an accelerator actually executes: SMs and warps, the memory hierarchy, tensor cores, the roofline, numerics, interconnects, and how TPUs, Trainium and MI300-class parts differ.
START HERE
01GPU Execution ModelFree
A GPU hides memory latency with parallelism instead of caches: thousands of threads in flight, scheduled in warps of 32, pinned to streaming multiprocessors that switch between warps for free whenever one stalls. Every performance conversation in an AI infra loop, from occupancy to why decode is slow, rests on this one mechanism.02GPU Memory HierarchyFree
A GPU has four places a byte can live, and they differ by a thousandfold in bandwidth: registers, shared memory on the SM, a chip-wide L2, and HBM off-chip. Almost every kernel optimization is a decision about which level a value is read from and how many times. Knowing the sizes and bandwidths for an H100 cold is what lets you say why a kernel is slow before you profile it.03Tensor Cores and Matrix UnitsCore
Tensor cores are fixed-function units that compute a small matrix multiply-accumulate per instruction, and they are where almost all of a modern GPU's FLOPS live: 989 dense bf16 TFLOPS on an H100 against about 67 from the general-purpose lanes. Only dense, well-shaped matrix multiplication at a supported precision can use them, which is why GEMMs reach peak and nothing else does, and why precision choices are throughput choices.04Roofline ModelFree
The roofline plots a kernel's attainable throughput against its arithmetic intensity, FLOPs per byte moved from memory. Below the ridge point (peak FLOPS divided by memory bandwidth, about 295 on an H100 in bf16) a kernel is memory-bound and no amount of clever code reaches the peak; above it, compute is the limit. One picture explains why decode runs at under 1% of peak and why fusion and batching are the two levers that move it.05Memory-Bound vs Compute-Bound KernelsPremium
Every kernel is limited by one of two walls: how fast bytes arrive from HBM, or how fast the tensor cores can multiply. Which wall applies is decided by arithmetic intensity against the ridge point, and the two regimes need opposite fixes. Decode, LayerNorm and softmax are memory-bound; prefill GEMMs are compute-bound; the interview question is which one you are looking at and what you would do about it.06Numerics: FP32, BF16, FP8 and FP4Core
Every number format is a trade between range (exponent bits), precision (mantissa bits) and throughput (fewer bits, more values per cycle through the tensor cores). bf16 won training because it keeps fp32's range; fp8 splits into E4M3 for precision and E5M2 for range and needs scaling factors; fp4 needs block scaling and careful outlier handling. Knowing which format goes where, and why accumulation stays fp32, is what the numerics question is really asking.07NVLink, NVSwitch and PCIePremium
Inside a node, GPUs talk over NVLink at 900 GB/s per H100 through an NVSwitch fabric that gives all eight cards full bandwidth to each other; to the host and to anything outside the node they talk over PCIe at 64 GB/s or a 400 Gb/s NIC at 50 GB/s. That fifteen-fold gap is why tensor parallelism stays inside the eight-GPU domain, why NVL72 changes the serving math for MoE, and why the question "how many GPUs share an NVLink domain?" is the first thing to ask about any cluster.08GPU Generations: A100 to BlackwellPremium
Four NVIDIA generations are in fleets at once, and interviewers ask what each one changed, not what it is called. A100 to H100 added fp8 and tripled compute; H200 kept the die and grew memory; B200 doubled everything and added fp4; B300 stacked more HBM and cut fp64. This page carries the dense numbers for each, what they did to training and serving, and the marketing traps (sparse peaks, 192 versus 180 GB, die counting) that trip candidates. Dated September 2026.09TPU Architecture and Systolic ArraysPremium
A TPU is a matrix unit first and a processor second: a systolic array that streams activations through a grid of multiply-accumulate cells holding stationary weights, fed by a compiler rather than a warp scheduler, and scaled out over a dedicated inter-chip interconnect into pods of thousands. The roofline thinking transfers from GPUs unchanged; the workflow does not, which is what the Google loop tests.10Trainium and InferentiaPremium
AWS's accelerators trade the GPU's general-purpose flexibility for a compiler-driven design with separate tensor, vector, scalar and GPSIMD engines, software-managed on-chip SRAM, and a proprietary NeuronLink fabric. Trainium2 delivers 667 dense bf16 TFLOPS with 96 GB at 2.9 TB/s, Trainium3 about the same bf16 with 2.5 PFLOPS of fp8 and 4.9 TB/s. The pitch is cost per FLOP; the price is a kernel ecosystem you may have to build yourself, which is exactly what the AWS loop probes.11AMD Instinct and ROCmPremium
AMD's Instinct line competes on memory: 192 GB on the MI300X, 256 on the MI325X, 288 on the MI355X, with dense fp8 peaks at or above NVIDIA's at each generation. The catch is software: ROCm and HIP run most PyTorch and the main serving engines, but the attention, MoE and quantization kernels arrive on CUDA first and measured throughput has trailed the spec sheet. Interviewers want the numbers, the wavefront-of-64 difference, and a clear-eyed statement of when the memory argument wins.12Cerebras, Groq and Dataflow AcceleratorsPremium
Cerebras and Groq attack the same wall from the same direction: keep the working set in on-chip SRAM, which is tens of times faster than HBM, so that memory-bound decode stops being memory-bound. Cerebras does it with a wafer-scale engine holding 44 GB of SRAM; Groq with a deterministic, compiler-scheduled dataflow chip holding a few hundred megabytes per part and many parts per model. Both win single-stream decode by large factors and pay in capacity, cost per token at high batch, and flexibility. Knowing where each fits is the interview question.02
🖧 Hardware & Cluster Build-Out
The physical layer: which accelerator for which workload, SXM against PCIe against rack-scale, NVLink domains, scale-out fabrics, cables and optics, rack power, liquid cooling, the bill of materials, bring-up and acceptance.
01Accelerator Selection: H100 to B300 and RTX PRO 6000Free
Three published numbers decide which accelerator suits a workload, and they are independent: memory capacity gates what fits, memory bandwidth gates decode speed, and tensor FLOPS gate prefill and training. As of September 2026 the parts NVIDIA sells for datacenters span 80 GB to 288 GB and 1.6 TB/s to 8 TB/s, and the gap between the compute number and the bandwidth number has widened every generation, which is why a part that looks four times faster on a slide is often twice as fast on a decode workload.02SXM, PCIe and Rack-Scale Form FactorsFree
The same silicon ships in three shapes and the shape decides the deployment. An SXM module is soldered to a baseboard with a full NVLink mesh and needs 700 to 1,400 W of direct power and usually liquid cooling. A PCIe card slots into a standard server, draws through the slot and a cable, and has no NVLink. A rack-scale system like GB300 NVL72 makes the whole rack one NVLink domain and stops being a server at all. Choosing between them fixes your power, cooling, cabling and scheduling story.03NVLink Domains and the NVL72 RackFree
An NVLink domain is the set of GPUs that can address each other's memory at full fabric speed, and its size is the single most consequential number in a cluster design. Eight on an HGX node, 72 on a GB300 NVL72 rack. Inside the domain a collective moves at terabytes per second over a copper backplane; outside it, the same collective drops to the scale-out fabric at 800 Gb/s per GPU, a gap of roughly twenty times that decides how models are sharded.04Scale-Out Fabric Choice: InfiniBand XDR vs Spectrum-XFree
Outside the NVLink domain every GPU talks over a scale-out fabric, and as of September 2026 NVIDIA sells two at the same 800 Gb/s per port: Quantum-X800 InfiniBand and Spectrum-X Ethernet. They differ in congestion handling, operational familiarity and what happens when something misbehaves rather than in headline speed. The switch radix decides how many endpoints a two-tier fabric reaches, and that single number drives the switch count, the cable count and a large part of the budget.05Cables, Transceivers and the Optics Power BudgetFree
Cable choice is set by distance and it is the most common ordering mistake in a GPU cluster build. Passive copper reached 3 m at 400G and tops out near 2 m at 800G, so a bill of materials copied from the previous generation produces links that will not come up. Beyond copper come active copper, then active optical cables, then transceivers and fiber. Each step adds reach and adds power, and at cluster scale the transceivers alone draw tens of kilowatts that nobody budgeted.06Rack Power Delivery and BuswaysFree
A GPU rack has gone from 10 kW to over 120 kW in a few generations, and the electrical design changed with it. At 132 kW on a 415 V three-phase feed a rack draws about 184 amps, which is past what a normal power strip carries, so distribution moves to overhead busway and the rack takes redundant high-current taps. On top of the steady draw sits a synchronized transient every training step, because thousands of GPUs finish a collective at the same instant, and that swing is what sizes the upstream equipment.07Direct-to-Chip Liquid Cooling and CDUsFree
Above roughly 40 kW a rack cannot be cooled by air in any practical hall, which is why every dense GPU deployment now runs liquid to the chip. A cold plate sits on each GPU, a coolant distribution unit isolates the clean rack loop from facility water, and the facility side runs warm, typically 30 to 40 degrees supply, because warm water is cheaper to make. The design numbers are flow rate and temperature rise, and both fall out of one equation that every operator should be able to do from memory.08The Bill of Materials for a Training ClusterFree
A GPU cluster is not a pile of GPUs. A 512-GPU scalable unit built to NVIDIA's DGX SuperPOD B300 reference architecture needs 64 nodes, four separate networks, thousands of transceivers, storage that can absorb a checkpoint burst, a management plane, racks, power distribution and cooling equipment. Writing the list out in order is how a design becomes a purchase order, and the items people forget are the ones that hold up a deployment for weeks.09Cluster Bring-Up: Firmware, Drivers and the StackFree
Bring-up is an ordered dependency chain and skipping a step produces a symptom that points somewhere else. Firmware first, then the operating system and kernel, then the GPU driver, then the fabric manager, then the network stack, then GPUDirect, then CUDA and NCCL, then the container and scheduling layer. On Blackwell HGX systems the fabric manager reaches the NVSwitches through a bridge device and therefore depends on the InfiniBand stack being present, which is a dependency that surprises almost everyone the first time.10Burn-In and Acceptance TestingFree
New hardware fails early or it fails late, and burn-in exists to move the early failures before the cluster is handed over rather than after. A proper acceptance test runs every layer under sustained load for days, compares every node against its siblings rather than against a specification, and produces a signed number the buyer and the vendor both agree on. The comparison is the important part: identical hardware running identical work should produce identical numbers, and the outliers are the finding.11Colocation, Power Contracts and Site SelectionFree
For most organizations the constraint on deploying GPUs is not the GPUs. It is finding a hall that can deliver 100 kilowatts or more per rack, reject that heat with liquid, and sign a contract for the power years before the hardware exists. Colocation contracts price reserved capacity rather than consumption, cooling capability is what eliminates most sites, and the lead time on new electrical supply is measured in years while GPUs arrive in months.03
⚡ Kernels & Compilers
Why kernels are fast or slow: coalescing, shared memory, occupancy, fusion, tiling, FlashAttention, Triton, and the profiler view that explains all of it.
01CUDA Programming ModelFree
CUDA splits a program into a host that allocates, copies and enqueues work, and a device that runs thousands of identical threads organized as a grid of blocks. Getting the split right, and knowing that a launch returns before the kernel runs, decides whether your first live-coding kernel produces a correct number or a silent zero.02Memory CoalescingCore
A warp's 32 threads issue one memory request together, and the hardware serves it in 32-byte sectors. Coalescing is arranging addresses so those sectors are full of bytes the warp will use. It decides whether a bandwidth-bound kernel moves at the HBM rate or at an eighth of it, and it is the pattern NVIDIA's trace-classification interview question tests.03Shared Memory and Bank ConflictsPremium
Shared memory is the programmer-managed SRAM inside each SM, split into 32 four-byte banks that serve one word each per cycle. When several lanes of a warp hit the same bank at different addresses the access serializes, and a 32-way conflict makes a shared-memory-bound loop run over ten times slower. Padding, XOR swizzles, cp.async and TMA are the tools that decide whether a tiled kernel gets the bandwidth it staged data for.04Occupancy and Register PressurePremium
Occupancy is the fraction of an SM's 64 warp slots that are resident, and it is capped by the 65,536 registers and 228 KB of shared memory each block consumes. It decides how much memory latency the hardware can hide for free, but the fastest kernels on a GPU routinely run at 25 percent, so the interview skill is knowing when to raise it and when to stop.05Kernel FusionFree
An elementwise or reduction kernel does a few FLOPs per byte and runs at HBM speed, so a chain of five of them costs five trips through HBM for work that needs one. Fusion collapses the chain into a single kernel that keeps intermediates in registers. It is the first lever for anything memory-bound, and knowing what it cannot fix (weight reads in decode, the GEMMs themselves) is what the interview is really testing.06Tiled Matrix MultiplicationPremium
A matrix multiply has enough reuse to be compute-bound, but only if the kernel captures that reuse in shared memory and registers instead of re-reading HBM. Tiling is how: a block owns an output tile, streams K-slices of A and B through shared memory, and each thread accumulates a small register tile. It is the live-coding exercise that separates people who know the roofline from people who have climbed it.07FlashAttention InternalsPremium
Standard attention writes the N x N score matrix to HBM and reads it back, which makes it memory-bound and quadratic in memory. FlashAttention tiles Q, K and V through shared memory, keeps a running max and sum so the softmax never needs the full row, and recomputes scores in the backward pass. Knowing the online-softmax rescale, why FlashAttention-2 flipped the loop order, and what FlashAttention-3 overlaps on Hopper is the difference between naming the paper and being able to write the kernel.08Triton Programming ModelCore
Triton replaces CUDA's thread with a program that owns a whole block of data, and replaces manual shared-memory staging and coalescing with a compiler that derives them from block shapes. Pointer arithmetic on vectors, masks for the tail, and program-id swizzling for L2 reuse are the three idioms every Triton kernel is built from, and the live exercise at Anthropic, OpenAI and the serving startups is usually one of a fused softmax, a LayerNorm or a matmul in exactly this style.09CUTLASS and Tensor Core KernelsPremium
CUTLASS is NVIDIA's template library for building GEMM-shaped kernels that run tensor cores at near cuBLAS speed while letting you change the data types, the tile shapes and the epilogue. Its hierarchy (device, kernel, collective mainloop, tile, instruction) is the vocabulary of every tensor-core discussion, and knowing when it beats calling cuBLAS or writing Triton is the judgment question kernel interviews end on.10Profiling with NsightPremium
Nsight Systems answers where wall-clock time goes across CPU, kernels and copies; Nsight Compute answers why one kernel is slow, from hardware counters. The skill interviewers test is the order: timeline first, then the Speed of Light section, then the two or three metrics that name the bottleneck, so that a memory-bound kernel is recognized from its profile in under a minute and the fix is bytes, not occupancy.11torch.compile and CUDA GraphsPremium
torch.compile captures Python into a graph with Dynamo, fuses it into Triton kernels with Inductor, and can wrap the result in a CUDA graph so a whole forward pass is one launch. CUDA graphs are what make batch-1 decode fast in every serving engine, and graph breaks, recompiles and static-shape rules are what make both bite in production. Interviewers ask when compile helps, when it hurts, and how you would know.04
🕸️ Distributed Training
Every parallelism and what it costs: data, ZeRO/FSDP, tensor, pipeline, context, expert, the collectives underneath, MFU, and surviving failures at scale.
01Data Parallelism and DDPFree
Data parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.02ZeRO and FSDPCore
ZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.03Tensor ParallelismPremium
Tensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.04Pipeline Parallelism and the BubblePremium
Pipeline parallelism puts consecutive groups of layers on different GPUs and streams micro-batches through them, which is the only parallelism whose traffic is small enough to cross a slow fabric comfortably. Its cost is the bubble, the idle time while the pipeline fills and drains, and the schedule you pick (GPipe, 1F1B, interleaved, zero-bubble) decides how much of each step is wasted.05Context and Sequence ParallelismPremium
Context parallelism splits a single long sequence across GPUs so that attention over 128k tokens fits in memory that would otherwise need terabytes of activations per layer. Ring attention rotates key-value blocks around the ranks while queries stay put, and grouped-query attention is what makes that rotation cheap enough to hide behind the attention math.06Expert Parallelism for MoEPremium
A mixture-of-experts layer runs only a few of its experts per token, so the experts can be spread across GPUs and each token shipped to the ranks that hold its chosen experts. That shipping is an all-to-all in each direction, twice per layer per pass, and its cost plus the load imbalance between experts is what expert parallelism is really about.07Collective Communication PrimitivesFree
All-reduce, all-gather, reduce-scatter, all-to-all and broadcast are the five operations every parallelism strategy is built from, and each has a fixed per-rank traffic cost you can compute before a job runs. Knowing those volumes for a named model is how you decide whether a layout is compute-bound or waiting on the network.08Ring vs Tree All-ReducePremium
A ring all-reduce moves the minimum possible bytes per rank but takes 2(N-1) steps, so its latency grows with the number of GPUs; a tree finishes in a logarithmic number of steps but is harder to keep bandwidth-optimal. NCCL keeps both, chooses per message size and rank count, and reading its choice is how you diagnose a collective that is slower than the fabric allows.09MFU and HFUCore
Model FLOPs utilization is the fraction of a GPU's peak that goes into the model's own forward and backward math, computed from 6ND and the step time; hardware FLOPs utilization also counts recomputation. Production LLM training lands at 35 to 45% MFU, and knowing where the other 55% goes is the job.10Activation CheckpointingPremium
Activation checkpointing discards a layer's intermediate tensors during forward and recomputes them during backward, trading up to a third more compute for an order-of-magnitude cut in activation memory. Selective recomputation keeps the tensors that are expensive to recompute and cheap to store, and the extra FLOPs show up as HFU without adding a token of progress.11Checkpointing and Resumption at ScalePremium
A training checkpoint at frontier scale is terabytes of sharded optimizer state that must be written often enough to bound lost work and fast enough not to stall the job. The interval is a formula in the failure rate and the write cost, and asynchronous sharded writes are what turn it from a 15% tax into a 3% one.12Elastic and Fault-Tolerant TrainingPremium
At 16,384 GPUs something fails every few hours, so a training job is a loop of detect, isolate, replace and resume rather than one process that runs for months. The Llama 3 record of 466 interruptions in 54 days is the reference point, and the design goal is keeping effective training time above 90% by making each event cost minutes.13RL Post-Training InfrastructurePremium
Reinforcement-learning post-training alternates between generating long responses with an inference engine and updating the policy with a training framework, and the two phases want different hardware, different parallel layouts and a copy of the same weights each. The infrastructure problem is keeping the generators busy while the learner trains, syncing terabytes of weights between them every step, and surviving the long tail of response lengths.05
🚀 Inference & Serving
From a forward pass to a serving platform: prefill and decode, the KV cache, batching, paging, speculation, disaggregation, quantization and the engines that ship them.
01Prefill vs DecodeFree
An LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.02The KV CacheFree
The KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.03Continuous BatchingCore
Continuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.04PagedAttentionPremium
PagedAttention stores the KV cache in fixed-size blocks scattered across HBM and maps each sequence's logical positions to physical blocks through a block table, the same trick an operating system uses for virtual memory. It removes the reservation and fragmentation waste of contiguous allocation, lets blocks be shared between sequences, and is why an engine can decide admission by counting free blocks.05Chunked PrefillPremium
A long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.06Speculative DecodingPremium
Decode is memory-bound: each step reads every weight to produce one token. Speculative decoding has a cheap draft propose several tokens, then verifies them all in one forward pass of the big model, so one weight read yields several tokens with output distribution unchanged. It wins 2x to 3x at small batch, breaks even near the ridge point where the GPU is already compute-bound, and lives or dies on the acceptance rate, which is what interviewers ask you to reason about.07Disaggregated Prefill and DecodePremium
Prefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.08Prefix Caching and KV ReusePremium
Most requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.09Attention Variants: MHA, GQA, MQA and MLACore
The KV cache scales with the number of key-value heads, and the four attention variants differ exactly there: multi-head keeps one KV head per query head, multi-query keeps one for all, grouped-query shares one across a group, and multi-head latent attention caches a compressed latent instead of keys and values at all. For Llama 3.1 70B that is the difference between 2.6 MB and 320 KB per token; for DeepSeek-V3 it is about 70 KB. The variant a model was trained with is a serving decision made before the first GPU was bought.10Quantization for InferencePremium
Quantization stores weights, and sometimes activations and the KV cache, in fewer bits, which cuts the bytes a decode step has to stream and the memory a model occupies. Weight-only int4 (GPTQ, AWQ) is a capacity and single-stream latency play; fp8 for weights and activations (W8A8) doubles tensor-core throughput and helps prefill and large batch; fp8 KV cache doubles context per GPU. Each has an accuracy cost you measure rather than assume, and knowing which one to reach for from the bottleneck is the interview question.11Serving Engines: vLLM, SGLang and TensorRT-LLMPremium
Three engines serve most open-weight models in production, and they converged on the same mechanisms (paged KV, continuous batching, chunked prefill, prefix caching, speculation, disaggregation) while differing in what they optimize first. vLLM is the default for breadth and hardware coverage, SGLang leads on prefix reuse and structured generation, TensorRT-LLM squeezes the most from NVIDIA hardware at the cost of a compile step. This is a dated page, September 2026; the decision table is what to carry into the room, not the version numbers.12Latency Metrics: TTFT, TPOT and GoodputFree
An LLM request has two latencies, not one: time to first token, set by queueing and prefill, and time per output token, set by the decode loop. Reporting them as percentiles, and reporting goodput (requests that met both SLOs per second) rather than raw throughput, is what separates a serving engineer from a benchmark reader. The numbers a loop expects: about 24 tokens per second single-stream for a 70B model on one H100, TTFT floors in the hundreds of milliseconds for long prompts, and p99s that come from queueing, not from the GPU.13Multi-LoRA ServingPremium
A LoRA adapter is a few hundred megabytes of low-rank matrices that turn a base model into a fine-tuned variant, and multi-LoRA serving runs hundreds of them on one copy of the base weights by keeping the adapters in memory and applying the right one per request inside the batch. It is how a platform serves a thousand customers' fine-tunes without a thousand deployments. The costs are an extra small matmul per layer, adapter memory and loading, and a scheduler that has to batch across adapters without starving any of them.14Inference Autoscaling and Cold StartsPremium
Scaling an LLM fleet is harder than scaling a web service because a replica takes minutes to become useful (pull an image, load 141 GB of weights, warm the cache) and costs several dollars an hour while idle. The signals that work are queue depth and TTFT against the SLO, not GPU utilization, which is misleading for memory-bound decode. The design is a warm pool sized for the burst, hysteresis so the fleet does not thrash, and a cold-start path measured in seconds through snapshots and weight streaming.06
🧮 Open Weights & Serving Engines
Serving the 2026 open-weights models: sizing from config.json, latent attention and sparse indexers, vLLM and SGLang arguments, expert parallelism, weight formats, multi-node topologies, benchmarking and capacity planning.
01Open-Weights Models of 2026Free
The open-weights frontier moved from dense models of tens of billions of parameters to sparse mixtures of experts measured in trillions, and the serving problem changed with it. As of September 2026 the releases an infrastructure engineer is asked about are Z.ai's GLM-5.3 at 753B, Moonshot's Kimi K3 at 2.8T, and DeepSeek's V4 family. What matters for deployment is not the headline count but three other numbers: active parameters per token, the attention design, and the format the weights actually shipped in.02Reading config.json to Size a Model You Have Never RunFree
Every Hugging Face model ships a config.json, and it contains enough to compute the weight footprint, the KV cache per token, the parallel degrees that divide cleanly and the minimum GPU count, before downloading a byte. Doing that derivation is a standard whiteboard exercise in serving interviews because it is exactly what an engineer does on the morning a new model lands, and the fields that matter are the same across every recent architecture.03Multi-Head Latent Attention and Sparse IndexersFree
Grouped-query attention shrank the KV cache by sharing key and value heads. Latent attention goes further by caching a single compressed vector per token per layer and reconstructing the heads on the fly, which cuts the cache by tens of times rather than by a small factor. On top of that, sparse indexers pick a few thousand relevant positions per query instead of attending to all of them, turning the quadratic term linear at long context. Both are now standard in open-weights models, and both change how a serving deployment is sized.04vLLM Server Arguments That MatterFree
A vLLM deployment is mostly decided by a dozen flags, and the ones that matter fall into four groups: how the model is split across GPUs, how memory is divided between weights and cache, how requests are batched, and which specialized backends the model needs. Getting the first two wrong produces an engine that will not start or that runs out of memory under load. Getting the third wrong produces an engine that starts, serves, and misses its latency target by a wide margin.05SGLang Server Arguments That MatterFree
SGLang's tuning model is different from vLLM's in one way that matters: it exposes the scheduler's aggressiveness and the static memory fraction as direct knobs, and its own documentation gives target values for the runtime signals those knobs move. That makes tuning it a measurement loop rather than guesswork. Aim for a queue of a hundred to a couple of thousand requests, token usage above 0.9, and five to eight gigabytes of free GPU memory after startup, then adjust the flags that move each one.06Expert Parallel and All-to-All BackendsFree
A mixture-of-experts model can be split two ways and the choice changes everything. Tensor parallelism shards each expert across GPUs, which keeps every GPU busy and reads every expert's shard on every token. Expert parallelism gives whole experts to whole GPUs, which reads only the selected experts but requires an all-to-all to route tokens to them and back. The all-to-all is the cost, its backend is a configuration choice matched to the interconnect, and expert load imbalance is what actually limits the result.07Weight Formats: FP8 Blocks, MXFP4 and AWQFree
Open-weights models now ship pre-quantized, and the format is part of the release rather than something you choose afterwards. Block-scaled FP8 gives one byte per parameter with a scale per tile. MXFP4 gives about 0.53 bytes by pairing four-bit values with a shared exponent every 32 elements. Integer schemes like AWQ reach similar sizes with a different error profile. What decides a deployment is not which is most accurate in the abstract but which one the model was released and evaluated in, and which one your engine and hardware can execute natively.08Multi-Node Serving TopologiesFree
Once a model needs more GPUs than one NVLink domain holds, the deployment shape becomes a real design decision. Tensor parallelism stays inside the node because it communicates twice per layer per token. Across nodes the choices are data parallelism with replicas, pipeline parallelism with a bubble, expert parallelism with an all-to-all, or disaggregation that runs prefill and decode on separate pools and ships the KV cache between them. Each has a different failure mode and a different scaling story.09Serving Benchmarks That Do Not LieFree
Most published serving numbers are not comparable to each other and not predictive of production, because they differ in the input distribution, the concurrency, whether the cache was warm, and which of several very different metrics is being reported. A benchmark that supports a decision has to fix all four, report a distribution rather than a mean, and be run against the traffic shape you actually serve. The single most useful discipline is to compute the bandwidth bound first, so you know what fraction of the possible you achieved.10Capacity Planning for Open-Weights FleetsFree
Planning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.11Model Onboarding: From Hugging Face to ProductionFree
A new open-weights model lands and someone asks how long until it is serving traffic. The answer depends on a sequence that is the same every time: read the card and the config, check engine support for the exact attention and quantization combination, size it, pull the weights, bring up one replica, validate correctness against the authors' own outputs, benchmark, then roll out behind a flag. The steps that surprise people are the download, which is hours for a trillion-parameter model, and the correctness check, which almost nobody does and which catches the wrong template.07
🧮 Napkin Math & Capacity
The formula sheet as concepts: footprints, FLOPs, intensity, bandwidth bounds, communication volume, cost per token, and how to size a cluster in your head.
01Model Memory FootprintFree
The first calculation in almost every AI infra loop: how many bytes does this model occupy, for inference and for training, and does it fit on the card in front of you? Inference is parameters times bytes per parameter (2 in bf16), plus a KV cache that grows with users. Training is 16 bytes per parameter before activations. A 70B model is 141 GB to serve and 1.13 TB to train, and a reader who can produce those two numbers from the parameter count, with the reasoning, has passed the first five minutes of the estimation round.02KV Cache SizingFree
The KV cache is the memory that decides how many users a serving replica can hold and how long their context can be. Its size per token comes from four numbers in the model's config file (layers, KV heads, head dimension, bytes per element) and one formula; multiplied by context and concurrency it is the number every capacity plan is built on. This page derives it, works it for four models including an MLA one, and shows the two places candidates get it wrong by a factor of eight.03Training FLOPs: 6NDFree
The compute needed to train a language model is six floating-point operations per parameter per token: two for the forward pass and four for the backward. Multiply by the parameter count and the token count and you have the whole run's compute, which is the number every fleet-sizing, time-to-train and cost question starts from. This page derives the 6, states the attention correction and when it matters, and shows where the 2N of inference comes from, so the reader can rebuild the formula rather than recall it.04GPU-Hours and Time to TrainCore
The fleet equation turns a training run's FLOPs into a schedule: time = 6ND divided by (GPUs times peak FLOPS times MFU). Every term is a stated assumption, and the interviewer grades the assumptions rather than the digits: which peak, which MFU, and what happens to the answer when MFU falls from 40% to 30%. This page works three runs end to end (an 8B, a 70B and a 405B), inverts the equation for the GPU count a deadline needs, and shows the sensitivity that separates a considered estimate from a lucky one.05Arithmetic Intensity by OperationCore
The roofline says a kernel's ceiling is set by its FLOPs per byte against the hardware's ridge point. This page does the FLOPs-per-byte arithmetic for the operations an LLM actually runs (decode at several batch sizes, prefill, the attention score matmul with and without FlashAttention, LayerNorm, an embedding lookup) so the reader can place any of them on the roofline from first principles and say which lever moves it. The numbers explain why a serving fleet's GPUs report 30% utilization while fully loaded.06Bandwidth-Bound Decode ThroughputPremium
Because decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.07Communication Volume EstimatesPremium
Every parallelism strategy is a promise to move a certain number of bytes between GPUs every step, and the fabric either affords it or it does not. This page derives the per-rank volume for data parallelism, ZeRO/FSDP, tensor parallelism, pipeline parallelism and expert parallelism, works each for a 70B model at 8 and 64 ranks, and turns the bytes into seconds on NVLink and on a 400 Gb/s NIC. The result is the rule that decides every 3D layout: per-layer traffic stays on NVLink, per-step traffic can cross the fabric.08Cost per Million TokensPremium
The unit every serving decision cashes out in. It is one formula: the fleet's dollars per second divided by the tokens per second it sustains, scaled to a million, with utilization in the denominator because idle replicas still cost money. This page derives it from a GPU price and a throughput estimate, works it at three batch sizes to show why batching is the main lever, separates prefill from decode pricing, and shows how the same fleet's cost per token moves by 5x between a quiet hour and a busy one.09TCO: Buy vs RentPremium
Whether to buy GPUs or rent them is a utilization question dressed as a finance question. An owned H100 costs a few tens of thousands of dollars up front and a known amount per hour in power, cooling, space and operations; a rented one costs a few dollars per hour and nothing when idle. The break-even is the utilization at which the owned hourly cost, amortized over the hardware's useful life, equals the rental rate. This page builds the owned cost from parts, works the break-even, and adds the terms the simple model leaves out: depreciation risk, reserved discounts, and the price of idle capacity.10Capacity Planning and UtilizationPremium
Capacity planning for GPUs is deciding how many to have next quarter given that they cost money whether busy or not, that demand arrives in bursts, and that a queue near saturation produces waits that grow without bound. This page works the planning arithmetic for a serving fleet (peak demand, headroom, the p99 penalty of running hot) and a training platform (job mix, queue wait, the value of a shared pool), and gives the queueing intuition that makes 70% look full. The number that decides everything is utilization, and it has a ceiling set by latency, not by hardware.11Power and Datacenter ConstraintsPremium
The binding constraint on new GPU capacity in 2026 is not chips or capital but megawatts: an H100 node draws about 10 kW, a GB200 NVL72 rack about 120 kW, and a 100,000-GPU cluster needs on the order of 150 MW with cooling. This page converts GPU counts to power, power to cooling and facility requirements, and both to cost, so a candidate can size a training hall from a power budget and explain why liquid cooling, PUE and the local grid decide where the next cluster goes.08
🔌 Networking & Storage
The fabric under the collectives: RDMA, InfiniBand and RoCE, topologies, congestion, GPUDirect, and the storage tiers that feed training and checkpoints.
01NCCL and Collective AlgorithmsFree
NCCL is the library every PyTorch collective lands in, and its choice of ring or tree, channel count and protocol decides whether an all-reduce runs at fabric speed or at a third of it. Knowing what NCCL_DEBUG=INFO prints, and which environment variable changes which decision, is the difference between tuning a cluster and guessing at it.02RDMA, InfiniBand and RoCEv2Core
Training across nodes moves hundreds of gigabytes per step, and a CPU-driven TCP stack cannot feed a 400 Gb/s link. RDMA lets a NIC write straight into a remote GPU's memory with no kernel and no copies, and it runs over two fabrics: InfiniBand, which is lossless by design, and RoCEv2, which is Ethernet made lossless by configuration. The choice is operational as much as technical, and the numbers that decide it are per-GPU bandwidth, the collective's volume, and who will debug a pause storm at 3 a.m.03Rail-Optimized and Fat-Tree FabricsPremium
A GPU cluster's network is built from two ideas: a fat tree (Clos) that gives every node a path to every other node with a chosen amount of oversubscription, and rail optimization, which wires GPU i of every node to the same leaf switch so the collectives that dominate training stay one hop away. Sizing one is arithmetic on port counts, and the interview question is usually that arithmetic: how many switches, what oversubscription, and where the NVLink domain ends and the fabric begins.04Congestion Control for AI FabricsPremium
Collective traffic is the worst case a network can see: hundreds of senders transmit to the same receiver at the same instant (incast), every flow is large and long-lived, and RDMA cannot tolerate a dropped packet. Congestion control is the set of mechanisms (PFC, ECN with DCQCN, adaptive routing, packet spraying) that keep queues from overflowing without stalling the fabric. On plain Ethernet a busy all-reduce can fall to about 60% of link rate; with a tuned control loop it holds above 90%. Reading the counters that show which one you have is the on-call skill.05GPUDirect RDMA and GPUDirect StoragePremium
By default a byte leaving a GPU for the network or the disk makes a detour through host memory, crossing PCIe twice and costing a CPU copy. GPUDirect RDMA lets the NIC read and write GPU memory directly, and GPUDirect Storage does the same for NVMe. The win is not raw bandwidth (PCIe is the ceiling either way) but the halving of PCIe traffic and the removal of the host as a bottleneck, which is what makes collectives run at NIC rate and checkpoints run at drive rate. When it is silently off, everything still works, at half speed.06Topology-Aware CommunicationPremium
The same collective can run at 900 GB/s or at 50 GB/s depending on which links it is laid across, so the mapping of parallel groups onto hardware is a performance decision, not a deployment detail. The rule: tensor-parallel groups inside the NVLink domain, data-parallel rings along rails, pipeline stages across the fabric, and every rank placed so its partner is one hop away. NCCL discovers the topology and does most of this when the job lets it; the failures come from placements that do not.07Parallel Filesystems vs Object StorageCore
A training cluster's storage has two very different jobs: stream terabytes of training data to thousands of GPUs at a steady rate, and absorb a multi-terabyte checkpoint burst every few minutes. Parallel filesystems (Lustre, GPFS, WEKA, VAST, FSx) give POSIX semantics and hundreds of GB/s of aggregate throughput; object storage (S3 and its equivalents) gives durability and cost at a fraction of the price with high first-byte latency. Almost every real cluster uses both, and the interview question is which job goes where and how big each tier has to be.08Data Loading Pipelines for TrainingPremium
The dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.09Checkpoint I/OPremium
A checkpoint of a 70B model's training state is 1.13 TB, and a 405B model's is 6.5 TB. Written synchronously every half hour, it stalls thousands of GPUs for as long as the slowest rank takes to reach the disk. The design that scales writes each rank's shard in parallel, copies it off the GPU to host memory first so the run resumes in seconds, and drains it to durable storage in the background. The arithmetic is bytes per rank against the bandwidth of each hop, and the goal is a checkpoint that costs the run under 1% of its time.10Debugging a Slow All-ReduceFree
A training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.11Dataset Lifecycle: Ingest, Shard and RetainFree
A training dataset is not a file, it is a pipeline with four stages and a retention policy, and each stage has a different bottleneck. Ingest is metadata-bound rather than bandwidth-bound. Tokenization is CPU work that should happen once offline rather than every epoch. Sharding decides whether the training read is a stream or a storm of small files. And retention decides how much of the bill is paid for bytes nobody reads.09
🗂️ Scheduling & Orchestration
How GPUs become a shared platform: Kubernetes device plugins and DRA, MIG and sharing, gang and topology-aware scheduling, Slurm, quotas, containers and cold starts.
01Kubernetes GPU SchedulingFree
Kubernetes knows nothing about GPUs until something tells it. The NVIDIA device plugin advertises each node's GPUs as a countable resource, the scheduler matches a pod's request to a node with enough of them, and the container runtime wires the device in. That model is enough for one job per GPU and breaks the moment you need sharing, topology or multi-node placement, which is where Dynamic Resource Allocation, the GPU Operator and the batch schedulers come in. Knowing which layer does what is the platform interview's opening question.02MIG, MPS and Time-SlicingCore
A whole H100 is far more than a notebook, a small inference service or a CI job needs, and giving each of them a card leaves most of the fleet idle. Three mechanisms share a GPU, and they differ in what they isolate: MIG partitions the hardware into up to seven slices with their own memory and compute, MPS lets several processes share one GPU's SMs concurrently with no memory isolation, and time-slicing context-switches between processes with no isolation at all. The choice is the isolation the workload needs against the utilization the platform wants.03Gang Scheduling with Kueue and VolcanoPremium
A distributed training job is 64 pods that start together or not at all: if 40 are running and 24 are Pending, the 40 hold their GPUs idle at a collective barrier waiting for ranks that may never come, and two such jobs can deadlock a whole cluster. Gang scheduling makes the job the unit of admission. Kueue and Volcano add queues, quotas, priorities and preemption on top, which is what turns a pile of GPUs into a platform several teams can share without starving each other.04Topology-Aware SchedulingPremium
Two placements of the same 64-GPU job can differ by 2x in step time: one keeps every tensor-parallel group on a single NVSwitch node and every data-parallel ring on a single rail, the other scatters ranks across racks and pushes per-layer traffic through the spine. The scheduler is the only thing that can prevent the second placement, because the framework maps ranks to whatever GPUs it is handed. Topology-aware scheduling means the scheduler knows the hierarchy (NVLink domain, rail, rack, spine block) and places gangs to keep traffic low in it.05Slurm for AI ClustersCore
Slurm is the scheduler most large training clusters still run, because it was built for exactly this shape of work: long jobs that need many nodes at once, launched with one command, placed with knowledge of the network. A candidate for a training-infrastructure role is expected to read an sbatch script, know how GPUs are requested and enforced, and explain why a job is stuck in the queue. This page covers the model, the commands that matter, the GPU-specific configuration, and the failure modes a platform engineer meets.06Slurm vs KubernetesFree
Every GPU platform team has this argument, and the two schedulers were built for different jobs: Slurm for long, large, all-or-nothing training on bare metal; Kubernetes for many services that scale up and down. Training fleets run Slurm because gang scheduling, topology and MPI-style launch are native there; serving fleets run Kubernetes because autoscaling and rolling deploys are native there. A platform that does both picks a hybrid: Slurm on Kubernetes (Slinky, Soperator) or a batch scheduler on Kubernetes (Kueue, Volcano, KAI). The interview question is which, for which workload, and why.07Multi-Tenancy, Quotas and Fair SharePremium
A shared GPU pool is cheaper than ten private ones because ten teams' demand is smoother than one team's, and it only works if the sharing is enforced. Quotas say what each team is guaranteed, borrowing lets idle guarantees be used by others, fair share decides who waits when everyone wants more, and preemption reclaims borrowed capacity. This page works the arithmetic that makes pooling worth it, the layers of isolation a tenant needs, and the incentive problems (hoarding, gaming, the research-versus-product tension) that any policy has to survive.08Ray on KubernetesPremium
Ray is a distributed compute framework that turns a cluster into one Python program: tasks and actors placed by Ray's own scheduler, with libraries for data loading, training, tuning and serving on top. KubeRay runs Ray clusters as Kubernetes resources. The combination is common in ML platforms because it gives researchers a single API for pipelines, distributed training and serving, and it is awkward in exactly one place: Ray has a scheduler and Kubernetes has a scheduler, and a platform engineer has to decide which one owns the GPUs. This page covers what Ray is good for, how KubeRay works, and where the two schedulers fight.09Containers, Images and GPU Cold StartsPremium
A GPU container is a 10 to 20 GB image whose CUDA libraries must match a host driver it did not ship with, that loads tens to hundreds of gigabytes of weights before it does anything, and that then spends a minute compiling and warming before the first request is fast. Every one of those steps is a cold-start cost, and the difference between a naive deployment (minutes) and a tuned one (seconds) is a chain of specific fixes: lazy image loading, driver compatibility done right, local weight caches, and snapshots of an initialized process. This page walks the chain with numbers.10Spot, Preemption and Capacity StrategiesPremium
Spot and preemptible GPUs cost a fraction of on-demand and can be taken back with a couple of minutes' notice, so using them well is an expected-value calculation: the discount against the work lost per preemption, which is set by checkpoint cadence and restart time. The same arithmetic governs internal preemption in a shared cluster. This page works the break-even, the checkpoint interval that makes spot pay, and the fleet mix (reserved baseline, on-demand headroom, spot for tolerant work) that a capacity strategy is built from.11Node Lifecycle: Drain, Upgrade and ReturnFree
A node moves through a fixed cycle between provisioning and decommissioning, and most fleet operations are one lap around it: cordon so nothing new lands, drain so running work finishes or moves, act, validate, then return to the pool. The wall-clock cost of a fleet-wide change is dominated by draining rather than by the change itself, which makes the plan a scheduling document rather than a technical one.10
🩺 Fleet Reliability & Observability
What breaks in a GPU fleet and how you see it: XID codes, ECC, NVLink, stragglers, thermal, health checks, SLOs, and the incident craft of keeping a training run alive.
01GPU Failure Modes and XID ErrorsFree
When a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.02DCGM and GPU TelemetryCore
NVIDIA's Data Center GPU Manager reads a GPU's counters, runs its diagnostics and exports both to the monitoring stack, and nearly every fleet's dashboards and alerts are built on it. The skill is knowing which of its hundreds of fields carry signal: the profiling metrics that say whether the tensor cores are busy (not the utilization number everyone reads first), the error counters that predict a failure, the throttle reasons that explain a slow step, and the diagnostic levels that decide whether a node returns to the pool. This page walks those fields, derives an MFU estimate from them, and gives a fleet's alert thresholds.03ECC, Row Remapping and Memory ErrorsPremium
HBM stacks flip bits, and the difference between a fleet that shrugs and one that loses a training step to corruption is error-correcting codes plus the machinery that retires bad memory before it produces a double-bit error. A single-bit error is corrected silently and counted; a double-bit error is detected, kills the process, and on Ampere and later triggers the row remapper to swap the failing row for a spare at the next reset. This page explains the codes, the remapper's states, how to read the counters as a prediction of failure, and the RMA rules a fleet applies.04NVLink and Fabric FaultsPremium
The links between GPUs are the part of a training node with the most connectors, the highest signalling rates and the least forgiveness: one marginal NVLink cable or one NVSwitch port turns an eight-GPU node into a straggler that slows a thousand-GPU job, and the symptom arrives as an NCCL timeout three layers away from the cause. This page covers what the links are, what their error counters mean, how a fault shows up in NCCL and in step time, how to isolate it to a GPU, a cable or a switch, and the arithmetic of why one degraded link is a whole-job problem.05Stragglers and HangsPremium
Synchronous training runs at the speed of its slowest rank, so one GPU that is 30% slow makes a thousand GPUs 30% slow, and one rank that never arrives at a collective makes the other 1,023 wait in silence until a watchdog fires ten minutes later. Finding the slow rank and the stuck rank is the most common on-call task on a training fleet, and the tooling for it (per-rank timing, the NCCL flight recorder, stack dumps across ranks) is specific and learnable. This page derives the straggler tax from first principles, lists the causes in the order they actually occur, and gives the procedure for a hang.06Node Health Checks and Burn-InPremium
The cheapest failure is the one caught before a job starts. A GPU fleet runs three kinds of check: a burn-in of days on new or repaired hardware to shake out the parts that die young, a pre-flight suite of minutes before a node is handed to a job, and continuous checks during the job. Each test has an expected number (a GEMM within a few percent of the fleet median, an all-reduce at rated bus bandwidth, a NIC at line rate, a memory test with zero errors), and a node that misses any of them goes to quarantine. This page gives the suites, the numbers, and the arithmetic that justifies the time they cost.07Training Uptime and Interruption StatisticsFree
A frontier training run is a months-long job on tens of thousands of parts, each of which fails rarely, so the run as a whole fails constantly. The numbers that describe this are simple and worth deriving from scratch: a per-GPU failure rate, multiplied by the fleet, gives the interval between stops; the checkpoint interval and the restart time give the work lost per stop; together they give effective training time, which is the fraction of wall-clock the run is actually learning. This page derives all of it from the one public dataset that reports it and shows how to size checkpointing so the run stays above 90%.08SLOs for AI SystemsCore
A service level objective is a promise with a number attached, and AI systems need their own because the classic ones do not fit: a training run has no requests, only progress, so its objective is goodput; an LLM endpoint streams, so its latency is two numbers (time to first token and time per token) rather than one; and both spend a budget that is set by hardware failure rates rather than by software bugs. This page defines the objectives that fleet and serving teams actually use, derives the thresholds from user needs and from the hardware, and works the error-budget arithmetic that decides when to stop shipping and start fixing.09Incident Response for GPU FleetsPremium
An incident on a GPU fleet is a training run that stopped, a serving endpoint burning its error budget, or a fleet-wide symptom nobody has explained yet. The response has a shape: detect, stabilize, diagnose, repair, return through the gate, write it up. The stabilizing move (drain the node, restart from checkpoint, or shift traffic) comes before the diagnosis, because a frontier run loses more per minute than any investigation is worth. This page gives the triage order, the 3am decision tree, the spare-capacity arithmetic behind drain-and-replace, and what a fleet postmortem has to contain.10Thermal, Power and Cooling EventsPremium
A GPU that gets too hot or is denied power does not fail; it slows down, and on a synchronous job a slow GPU is a slow job. Thermal and power events are the most common cause of the 'nothing failed but the run is 15% slower' ticket, and they are the incidents that scale from one node to a whole hall when a cooling distribution unit or a power feed has a problem. This page explains how throttling works, derives the step-time cost of a clock reduction, walks the failure modes of air and liquid cooling, and covers the power behaviour peculiar to training: thousands of GPUs going idle and busy in lockstep.11Alert Design and On-Call LoadFree
An alert exists to change what a human does, so any alert that fires without a decision attached is a false alarm regardless of whether its condition was true. GPU fleets generate a specific set of noisy signals that look serious and are not, and separating those from the ones that need a person at three in the morning is what keeps a rotation sustainable. The measure of an alerting system is the fraction of pages that led to an action.11
📐 AI Systems Design
The building blocks of the design round: routers, schedulers, queues, caches, autoscalers, multi-tenancy and the numbers that make a design credible.
01Inference Platform ArchitectureFree
An LLM inference platform is the layer between a product's API call and a GPU running a serving engine, and every design round starts from its reference shape: a gateway that authenticates and rate-limits, a router that picks a replica with the right model and a warm cache, a per-replica scheduler that batches, engines that run prefill and decode, a KV cache tier, an autoscaler, and the observability that makes it operable. This page draws that shape, sizes each box for a concrete workload, and walks the derivation from user demand to replica count that every design answer has to contain.02Request Routing and Load Balancing for LLMsPremium
A load balancer for stateless web services spreads requests evenly and is done. A router for LLM replicas has two things a web balancer never had to think about: each replica holds a cache (the KV pages of recent prefixes) that makes some replicas far cheaper than others for a given request, and each request costs a wildly different amount, so counting connections is meaningless. This page builds the router that handles both: prefix-aware placement with load-aware fallback, cost-aware queue estimates, session affinity, and the failure handling when a replica restarts and its cache is gone.03GPU Job Scheduler DesignCore
Design a scheduler for a shared GPU cluster is the most common design prompt in AI infrastructure interviews, because it touches everything: queues and priorities, gang placement, topology, fairness across teams, preemption and the checkpoints that make it survivable, and the failure handling that keeps a 512-GPU job alive. This page builds the design in layers, states the data model and the scheduling loop, derives the numbers (how long a job waits, how much preemption costs, how much fragmentation wastes), and lists the trade-offs the interviewer will push on.04Training Cluster Design at 10k GPUsPremium
Design a cluster for training frontier models is the prompt that tests whether a candidate can hold hardware, network, storage, scheduling and reliability in one head at once. The answer is a bill of materials with a reason for every line: how many GPUs and why, how they are grouped into pods, how the fabric connects the pods and what it costs a collective to cross one, how much storage bandwidth the checkpoints and the data loader need, how power and cooling bound the whole thing, and how the failure statistics set the spare pool and the checkpoint cadence. This page derives each line for a 10,240-GPU cluster.05Serverless GPU PlatformsPremium
A serverless GPU platform lets a customer deploy a function or a model and pay only while it runs, so the platform has to start a GPU workload in seconds, pack many customers onto shared hardware without letting them see each other, and keep enough capacity warm that a burst does not wait for a cold start. Each is a design problem with numbers: the cold-start chain and the snapshot that shortens it, bin-packing memory-sized workloads onto fixed-size GPUs, the isolation boundary and its cost, and the economics of idle capacity against cold starts. This page designs the platform and derives the trade-offs.06Multi-Tenant Fine-Tuning ServicePremium
A fine-tuning service takes a customer's dataset and a base model and returns a model, and the design problem is that many customers want this at once, cheaply, without seeing each other's data, on GPUs that must not sit idle between jobs. LoRA changes the shape: an adapter is a few hundred megabytes rather than a copy of the base, so many jobs can share a base in memory and many adapters can be served from one replica. This page designs the service end to end: the pipeline, the LoRA arithmetic that sets memory and cost, the isolation, the scheduler that packs jobs, and the serving path.07Evaluation and Data Pipeline InfrastructurePremium
Behind every model release is a pipeline that turns raw text into training shards and a harness that runs thousands of evaluation prompts against every checkpoint, and both are infrastructure problems with GPU-sized budgets. The data side is a batch system: dedup, filter, tokenize and shard petabytes with lineage. The eval side is a serving system in disguise: run a benchmark suite against a checkpoint in minutes, on shared GPUs, reproducibly, with results a researcher can trust. This page designs both, derives the compute and storage they need, and gives the reproducibility rules that separate a real harness from a script.08Designing for Latency SLOsCore
A latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.09Capacity and BackpressurePremium
A system that accepts more work than it can finish does not degrade gracefully; it degrades completely, because every request it queues makes every other request slower until all of them time out. The defence is backpressure: bounded queues at every stage, admission control that rejects early when the expected wait exceeds the budget, load shedding by priority, and clients that back off. For LLM serving the stages are the gateway, the router, the engine's queue and its KV pool. This page works the arithmetic of why unbounded queues kill p99, designs the bounds per stage, and the client contract that keeps it stable under overload.10The AI Infra Design Round PlaybookFree
The AI infrastructure design round is 45 to 60 minutes with one prompt (design a serving platform, a training scheduler, a 10k-GPU cluster, a fine-tuning service) and one interviewer whose job is to find the edge of what you know. The candidates who pass do the same things in the same order: pin the requirements and the numbers in the first five minutes, draw the reference shape, size it with a stated chain of arithmetic, pick two deep dives, and name the failure modes before being asked. This page gives that structure with a minute-by-minute plan, the numbers to bring in your head, and the mistakes that end the round early.11Control Plane and API Design for GPU PlatformsFree
Every GPU platform has a control plane, and its API is what the rest of the organization experiences as the platform. Three semantics decide whether it survives contact with a network: idempotent creation so a retried request does not launch a second job on sixty-four GPUs, cancellation modelled as intent because only the node agent can stop a running process, and cursor pagination that does not skip rows when work is created during a listing.12Multi-Region Serving and FailoverFree
Running inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.12
💻 Coding for Infra
The engineering craft infra screens reward: concurrency, backpressure, idempotency, intervals and schedulers, and the patterns behind the practical builds.
01The GPU Credit Scheduler PatternFree
The most widely reported coding problem in AI infrastructure loops is a small scheduler: accounts hold credits, jobs arrive with a cost and a priority, and you must decide which jobs run, in what order, without letting any account overspend, then extend it under follow-ups (refunds, reservations, concurrency limits, fairness). It is not a trick question; it is a test of whether you can model state cleanly, pick the right data structures, keep invariants under mutation, and talk about complexity while typing. This page works the problem from the first line to the fourth follow-up, with the code, the invariants, and the derivations.02Rate-Limiting AlgorithmsCore
A rate limiter answers one question, 'may this request proceed now?', and the three classic algorithms answer it with different shapes of fairness and memory: the token bucket allows bursts up to a capacity and refills at a rate, the leaky bucket smooths output to a fixed rate, and sliding windows count recent requests exactly or approximately. AI platforms limit in tokens as well as requests, per tenant, across many gateways, which adds two twists: a request's cost is unknown until it finishes, and the counters must be shared. This page derives each algorithm, implements the token bucket correctly, and covers both twists.03Batching Queues and BackpressurePremium
Write a request batcher is the coding round's version of the serving engine's scheduler: requests arrive one at a time, the GPU wants them in groups, and the batcher decides when a group is full enough to send without holding anyone too long or accepting more than it can hold. The two knobs are the maximum batch size and the maximum wait, the invariant is a bounded queue, and the follow-ups (priorities, cost-aware batching, cancellation, bounded in-flight batches) are the ideas the real engines carry. This page implements the batcher in asyncio, derives what each knob buys, and walks the follow-ups.04Retry, Backoff and IdempotencyCore
A retry is a second request that the system did not budget for, and a thousand clients retrying at the same moment is a second outage that the first one caused. The craft is small and specific: retry only what is safe to retry, wait an exponentially growing random interval so the retries spread out, cap the total retries with a budget, and make every retried operation idempotent so a duplicate does not double-charge or double-train. This page derives why synchronized retries double the load, works the jitter arithmetic, implements the client correctly, and covers idempotency keys for the operations an AI platform exposes.05Interval Merging and Utilization LogsPremium
Given busy intervals per GPU, when was the whole cluster idle? What was the utilization per hour from a log of start and stop events? Which jobs overlapped? These are the interval problems of the infrastructure coding screen, and they share one tool: sort the endpoints and sweep. The sweep line turns every variant into a single pass with a counter, the sort is the only thing that costs more than linear time, and the edge cases (touching intervals, zero-length events, an unterminated start) are where candidates lose the round. This page works the standard problem and its relatives with code, tests and the complexity derivation.06Concurrency in Python, Go and C++Premium
Infrastructure code is concurrent by nature: a loader feeding a GPU, a gateway holding ten thousand streams, a controller reconciling a fleet. The coding screen tests whether you know which primitive fits which problem in the language you claim, and the three languages the field uses answer differently: Python has one interpreter lock and an event loop, Go has cheap goroutines and channels, C++ has threads, mutexes and atomics with no safety net. This page gives the model of each, works the favourite problems (a thread-safe LRU, a worker pool, a bounded fan-out) in each, and derives when threads, processes or async buy throughput.07Producer-Consumer PipelinesPremium
A data loader, a log shipper, a batch inference job and a checkpoint writer are the same program: stages connected by bounded buffers, each running at its own pace, the slowest setting the throughput and the buffers absorbing the jitter between them. The coding screen asks you to build one (read, decode, batch, feed a consumer) and then pushes on the production questions: buffer sizes, clean stops, failure propagation, and why it runs at a third of the expected speed. This page derives throughput from stage times, implements the pipeline in threads and asyncio, and works the stop and failure semantics.08Parsing Kernel Traces and LogsPremium
The profiler exported a 40 GB trace; the fleet emitted a terabyte of logs overnight; the interviewer hands you a text file of kernel records and asks which kernels dominated, per GPU, per stream. The problem is a parser plus an aggregation, and it is a test of three habits: streaming instead of loading, choosing the key you aggregate on before you write a line, and handling malformed input as data rather than as an exception. This page works the reported trace-classification problem end to end, derives the memory bounds of each design, and shows the generator-based structure that scales from a screen-sized file to a fleet.09The Practical Coding Screen PlaybookFree
The AI infrastructure coding screen is 45 to 60 minutes of building a small, realistic piece of systems code (a scheduler, a rate limiter, a batcher, a log parser, a cache) in the language you choose, with an interviewer who extends the problem twice and watches how you handle it. It is not a puzzle round: the score comes from working code early, tests that name the invariants, complexity said out loud, and calm follow-ups. Some companies allow an AI assistant and some ban it, and each policy changes what is measured. This page gives the minute-by-minute plan, the habits that score, and the mistakes that end the screen.10Cache-Friendly Data StructuresFree
A cache line is 64 bytes and it is the unit of coherence, so where data sits decides how fast code runs more often than which algorithm it uses. Two consequences dominate infrastructure code: a lookup that chases a pointer pays two dependent memory stalls instead of one, and two threads updating adjacent variables contend for a line they do not logically share. Both are layout problems with layout fixes.11Consistent Hashing and ShardingFree
Splitting work across N servers with a modulo of N moves almost everything when N changes, which for a cache means throwing away almost all of it. Consistent hashing places servers and keys on a ring so adding or removing one moves only its share, and virtual nodes fix the imbalance a small ring otherwise has. In LLM serving the same structure routes requests by prompt prefix so a conversation reaches the replica already holding its cache.13
🧭 Ownership & Judgment
The half of the loop most engineers under-prepare: reliability pushback, on-call narratives, working with researchers, and the safety and mission conversations at the labs.
01The Reliability Pushback StoryFree
Every AI infra loop has a behavioral round, and the story it wants most is the one where you stopped something (a launch, a run, a hardware admission) because the data said to, and you were accountable for the cost of stopping. This page gives the skeleton that works: the situation, the signal you read, the decision and who owned it, the evidence you brought, and what changed afterward. It also gives the follow-up interviewers hold back, the version that sounds right and fails, and the line between a senior telling and a staff telling of the same story.02On-Call Narratives That LandCore
Every infrastructure loop has a round where you are asked to tell an incident story, and the interviewer is not listening for drama. They are listening for the signal you read, the decision you made under time pressure with incomplete information, the evidence you had for it, and what you changed afterward so the same page never fires again. This page gives the structure that makes an incident story land in four minutes, two worked narratives from GPU fleet and serving work, the follow-ups that test whether the story is real, the version that sounds heroic and fails, and what separates the senior telling from the staff telling.03Working with ResearchersPremium
Infrastructure engineers at AI labs and platform teams have an unusual customer: a researcher whose experiment is the company's product, who needs the cluster today, and whose request may be a bad idea for the fleet. The behavioral round tests whether you can serve that customer without being run by them: saying no with data, saying yes with conditions, finding the need behind the ask, and sharing ownership of outcomes neither side controls alone. This page gives the recurring situations at the boundary, the responses that work in each, worked narratives, and the answers that sound collaborative and fail.04Migrations and DeprecationsPremium
Every infrastructure career contains a migration nobody wanted: the scheduler swap, the driver upgrade across a live fleet, the storage move while training runs are in flight, the deprecation of the launcher every team's scripts depend on. The behavioral round asks about one because it tests the skills that matter most and show least on a résumé: sequencing under risk, keeping a rollback real, moving people who have no reason to move, and knowing when to stop. This page gives the shape of a migration story that lands, two worked narratives from GPU fleet work, and the answers that sound like leadership and fail.05Safety and Mission Rounds at the LabsFree
Several frontier labs include a conversation in the loop that is not about code: how you think about the risks of the technology, why you want to work on it here, what you would do if asked to build something you thought was unsafe. Candidates over-prepare a rehearsed position on AI risk when the round measures something simpler: whether you engage honestly, whether you can hold a view and its counterargument at once, and whether your reasons survive a follow-up. This page describes what these rounds test, the shape of answers that land for an infrastructure engineer, and the answers that sound safe and fail.06Your View on Where AI Infrastructure Is GoingPremium
Somewhere in a senior or staff loop an interviewer asks what you think happens next: to GPUs and their challengers, to training at scale, to inference economics, to the tools. It looks like small talk and it is scored. The answer that works is a thesis with a date on it, a reason grounded in numbers you can derive, the counterargument you find strongest, and the thing you would watch to know you were wrong. This page shows how to build such a thesis from the material on this site, gives three worked examples, and lists the answers that sound informed and fail.07Leveling Signals: Senior vs StaffPremium
The same loop decides two things: whether you are hired and at what level, and the second is decided by a small set of signals interviewers are trained to listen for. Senior is scope you own and execute; staff is scope you define, across teams, under ambiguity, with the organization's defaults changed behind you. The signals are audible in every round: how you frame a design prompt, which failure modes you name unprompted, whose problem an incident was, what changed after it, and what you chose not to do. This page lists the signals per round, the stories each level needs, and the down-leveling traps.08Deciding Under Incomplete InformationFree
Most infrastructure decisions are made before the evidence is complete, and the skill being assessed is not judgment in the abstract but classification: whether the decision can be undone cheaply. Reversible decisions deserve speed and a scheduled review; one-way decisions deserve the delay and a second opinion. Engineers who apply the same deliberation to both are slow where speed is free and careless where it is not.09Escalation That WorksFree
Escalation has a reputation as a political act because most of it is done badly: a problem handed upward with no options and an implicit request that someone else choose a side. Done well it is a one-page artifact with two or three costed options, a recommendation, the decision needed, a date, and what you will do by default if no answer arrives. That last line is what converts a message into a decision, and it is the part almost everyone omits.10Mentoring and Growing EngineersFree
Mentoring on an infrastructure team happens mostly under pressure, during incidents and reviews, where the instinct to take the keyboard resolves the problem faster and teaches nothing. The method that works is the mentee driving while the mentor asks questions, with a takeover condition agreed in advance so nobody negotiates it at two in the morning. It costs time, and choosing which situations can absorb that cost is the judgment being assessed.11Talking About Cost and Capacity with LeadershipFree
Infrastructure engineers are asked to justify large numbers to people who do not share their vocabulary, and the conversations go wrong in predictable ways: a technical objection with no alternative, a forecast with no assumptions, or a cost quoted in a unit the listener cannot act on. What works is a small number of costed options, a stated recommendation, the decision needed by a date, and every figure expressed in whatever the listener actually controls.