Browse 13 concept tracks

THE AI infra CURRICULUM

Understand the concepts before you drill the questions

Learn how GPU systems work, how models are served, and how training runs across machines. The explanations connect these ideas to worked examples and related interview questions. Follow a track in order, or look up a concept as you need it.

178 concepts across 13 tracks · foundational concepts are free · see them as a map

Begin the curriculum →

GPU & Accelerator Architecture

How an accelerator actually executes: SMs and warps, the memory hierarchy, tensor cores, the roofline, numerics, interconnects, and how TPUs, Trainium and MI300-class parts differ.

START HERE

01GPU Execution ModelFree

CUDA groups threads into warps and blocks that run on streaming multiprocessors. Learn how ready warps hide waiting, why registers limit residency, and when higher occupancy fails to improve performance.

02GPU Memory HierarchyFree

Trace storage scope and data reuse from registers through shared memory, caches and HBM. Worked register and traffic examples explain why a source-level load is not automatically an HBM transfer.

03Tensor Cores and Matrix UnitsCore

Tensor cores accelerate supported matrix multiply-accumulate operations. Their performance depends on precision, matrix shape and how efficiently the kernel supplies data.

04Roofline ModelFree

Derive a throughput ceiling from work, traffic and hardware limits. Read a measured point below the roof, distinguish memory boundaries, and choose an optimization without promising peak performance.

05Memory-Bound vs Compute-Bound KernelsPremium

Use a worked timing example to distinguish a bandwidth ceiling from a measured bottleneck. Learn which profiler evidence to inspect and why decode, prefill and a whole serving step need separate diagnoses.

06Numerics: FP32, BF16, FP8 and FP4Core

Floating-point formats trade range and precision for storage and compute cost. Understand exponent bits, significand bits, scaling and accumulation before choosing a format.

07NVLink, NVSwitch and PCIePremium

NVLink, NVSwitch and PCIe provide different GPU communication paths. Learn how their topology and bandwidth affect the placement of parallel work.

08GPU Generations: A100 to BlackwellPremium

GPU generations change memory, arithmetic support and interconnect capacity at different rates. Compare the specification that limits your workload rather than the generation name.

09TPU Architecture and Systolic ArraysPremium

TPUs use matrix units and compiler-managed execution to run machine-learning workloads. Memory, shapes and inter-chip communication still determine how efficiently a model runs.

10Trainium and InferentiaPremium

Trainium and Inferentia run workloads through AWS’s Neuron software stack. Evaluate memory, compiler support, interconnects and the cost of adapting the model.

11AMD Instinct and ROCmPremium

AMD Instinct GPUs offer substantial memory capacity and bandwidth. Workload performance also depends on ROCm, the model implementation and the kernels available for the chosen device.

12Cerebras, Groq and Dataflow AcceleratorsPremium

Dataflow accelerators organize computation and memory differently from GPUs. On-chip storage can help latency, but capacity, communication and batching determine the wider tradeoff.

13Tensors, Shapes and DtypesPremium

A tensor is an array of numbers with a shape, a representation and a storage location. Learn to read batch and sequence dimensions, calculate bytes, and distinguish a reshape from a transpose before sizing GPU work.

14TPU Generations: v4 to TPU 8Premium

TPU generations differ along three axes that decide a deployment: memory per chip, the ratio of compute to memory bandwidth, and how chips connect. Small-memory 2D-torus parts, large 3D-torus pods and the training/inference split in TPU 8 each fit different workloads.

15The TPU Software Stack: JAX, XLA, PyTorch and vLLM on TPUPremium

On a GPU you run eager PyTorch and call tuned kernels. On a TPU a compiler sees the whole program, so shapes, sharding and compile time become your main tools. This maps JAX, MaxText, PyTorch/XLA, TorchTPU, Pallas and vLLM TPU onto the CUDA stack for training and serving.

16Structured Weight SparsityPremium

Zeros accelerate inference only when a supported representation and kernel skip their work. Separate pruning quality, compressed storage, dispatch and end-to-end benefit.

Hardware & Cluster Build-Out

The physical layer: which accelerator for which workload, SXM against PCIe against rack-scale, NVLink domains, scale-out fabrics, cables and optics, rack power, liquid cooling, the bill of materials, bring-up and acceptance.

01Accelerator Selection: H100 to B300 and RTX PRO 6000Free

Choose an accelerator by checking memory capacity, memory bandwidth and compute throughput separately. Each limits a different part of the workload.

02SXM, PCIe and Rack-Scale Form FactorsFree

GPU form factors determine how accelerators connect, receive power and get serviced. Compare PCIe cards, SXM modules and rack-scale systems against the workload and the facility.

03NVLink Domains and the NVL72 RackFree

An NVLink domain connects a group of GPUs through a scale-up fabric. Its boundary helps determine where to place communication-heavy parallel groups.

04Scale-Out Fabric Choice: InfiniBand XDR vs Spectrum-XFree

InfiniBand and Spectrum-X connect GPUs across nodes. Compare congestion behavior, topology and operational requirements as well as port speed.

05Cables, Transceivers and the Optics Power BudgetFree

Cable selection depends on distance, port compatibility and power. Count both ends of optical links and check the exact part numbers before ordering.

06Rack Power Delivery and BuswaysFree

Rack power determines current, distribution equipment and redundancy requirements. Training workloads can also create load swings that average power hides.

07Direct-to-Chip Liquid Cooling and CDUsFree

Direct-to-chip cooling carries heat from cold plates to a coolant distribution unit. Flow, temperature, pressure and coolant quality all matter to the design.

08The Bill of Materials for a Training ClusterFree

A cluster bill of materials includes networking, storage, management, power, cooling and spares as well as GPUs. Size each from the workload and topology.

09Cluster Bring-Up: Firmware, Drivers and the StackFree

Bring-up validates the dependencies between firmware, drivers, networking and the runtime. Checking each layer makes faults easier to locate.

10Burn-In and Acceptance TestingFree

Acceptance testing checks that a new cluster meets agreed performance and reliability criteria. Sustained tests and comparisons between nodes help reveal defects before production.

11Colocation, Power Contracts and Site SelectionFree

A suitable site must deliver the required rack power, cooling and service capacity. Contract terms and facility lead times shape both cost and deployment dates.

Kernels & Compilers

Why kernels are fast or slow: coalescing, shared memory, occupancy, fusion, tiling, FlashAttention, Triton, and the profiler view that explains all of it.

01CUDA Programming ModelFree

CUDA programs launch GPU kernels from host code. Thread indexing, memory access and stream ordering determine whether the result is correct.

02Memory CoalescingCore

Coalescing combines a warp’s nearby memory accesses into fewer transactions. Inspect addresses across lanes to understand how much data each access actually fetches.

03Shared Memory and Bank ConflictsPremium

Shared memory lets threads in a block exchange and reuse data. Bank conflicts occur when an access needs multiple distinct words from the same bank.

04Occupancy and Register PressurePremium

Occupancy measures how many warps are resident relative to an SM’s limit. Registers, shared memory and launch shape constrain it, but higher occupancy is not always faster.

05Kernel FusionFree

Kernel fusion combines operations so intermediate results can stay on-chip. It can reduce memory traffic and launch overhead, but may increase resource pressure.

06Tiled Matrix MultiplicationPremium

Tiled matrix multiplication reuses input values across many output calculations. Shared-memory and register tiles reduce repeated data movement while consuming limited on-chip resources.

07FlashAttention InternalsPremium

FlashAttention computes exact attention in tiles without storing the full score or probability matrix in device memory. Follow the running softmax state, understand why inputs can still be reread, and distinguish the mechanisms introduced in the first three papers.

08Triton Programming ModelCore

Triton expresses GPU work as operations on blocks of values. You choose shapes, addresses and masks while the compiler maps the work onto hardware.

09CUTLASS and Tensor Core KernelsPremium

CUTLASS provides components for building GPU matrix kernels. It is useful when the required precision, batching or fused operations need more control than a library call offers.

10Profiling with NsightPremium

Nsight Systems shows where an application spends time; Nsight Compute investigates individual kernels. Use the timeline to choose what deserves detailed profiling.

11torch.compile and CUDA GraphsPremium

torch.compile optimizes captured computation, while CUDA graphs reduce repeated launch overhead. Both need validation against the model’s shapes, control flow and memory behavior.

12CUDA Compilation: PTX, Cubins and SASSPremium

CUDA source, PTX and native GPU instructions describe different stages of execution. Inspect the selected binary and its resource use before explaining a performance change.

13GEMM Work Partitioning and Wave QuantizationPremium

Output tiles determine how much independent matrix work is available. Split-K and work-centric schedules can reduce an underfilled final wave, while adding partial-result reduction and workspace costs.

14Parallel Prefix Scan and Stream CompactionPremium

A prefix scan gives every item the aggregate of earlier items. It turns per-item counts into disjoint output offsets and enables stable compaction, provided block carries and integer limits are handled correctly.

15Kernel Correctness ContractsPremium

A fast GPU kernel is useful only for inputs it handles correctly. Define shapes, strides, dtypes, numerical tolerance, storage ownership and synchronization before timing a replacement.

Distributed Training

Every parallelism and what it costs: data, ZeRO/FSDP, tensor, pipeline, context, expert, the collectives underneath, MFU, and surviving failures at scale.

01Data Parallelism and DDPFree

Replicas combine gradients from different data. Follow a two-worker update, fix unequal-token weighting, and see why queued gradient buckets can leave more than one reduction exposed.

02ZeRO and FSDPCore

ZeRO and FSDP reduce replicated training state by distributing it across workers. The memory saving comes with communication and temporary-buffer costs.

03Tensor ParallelismPremium

Tensor parallelism splits operations within a layer across GPUs. Derive the column/row matrix split, count training and inference collectives, and choose a group size from memory, latency and actual interconnect performance.

04Pipeline Parallelism and the BubblePremium

Pipeline parallelism assigns groups of layers to different devices. Microbatches keep stages busy, while the schedule controls idle periods and activation memory.

05Context and Sequence ParallelismPremium

Context parallelism distributes a long sequence across devices. Attention then exchanges the information each rank needs instead of keeping the whole sequence locally.

06Expert Parallelism for MoEPremium

Expert parallelism places different MoE experts on different devices. Routing tokens between them adds communication and makes uneven expert demand a performance concern.

07Collective Communication PrimitivesFree

Follow small vectors through all-reduce, reduce-scatter, all-gather and all-to-all. Separate the result each worker needs from the algorithm-specific byte count and measured completion time.

08Ring vs Tree All-ReducePremium

Trace one chunk through a ring, separate byte balance from startup rounds, and compare tree depth without assuming identical bandwidth. Use measured message sizes and topology to choose.

09MFU and HFUCore

Measure useful model work, separate recomputation, and align token counts, timing and hardware peaks. Learn what activity counters can and cannot tell you.

10Activation CheckpointingPremium

Activation checkpointing saves memory by recomputing selected forward results during backward. Choose what to retain by comparing storage cost with recomputation cost.

11Checkpointing and Resumption at ScalePremium

Learn what a training checkpoint must save, why an asynchronous snapshot needs isolation, and when a complete write becomes a safe restore point. Work through a 56 GB state budget and test the continuation.

12Elastic and Fault-Tolerant TrainingPremium

Recover the intended training run after a failure: fence old workers, restore a supported layout, reject stale checkpoint uploads and measure progress that survives. Includes a generation-check example and replay budget.

13RL Post-Training InfrastructurePremium

RL post-training connects response generation, scoring and policy updates. The infrastructure must coordinate model versions while handling variable response lengths and uneven phase costs.

14Forward Pass, Backpropagation and Optimizer StepsPremium

Follow one parameter through prediction, loss, gradient and update. Understand why backward does not itself train the weights, why training needs extra memory, and how microbatches differ from optimizer steps.

Inference & Serving

From a forward pass to a serving platform: prefill and decode, the KV cache, batching, paging, speculation, disaggregation, quantization and the engines that ship them.

01Prefill vs DecodeFree

Follow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.

02The KV CacheFree

The KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.

03Continuous BatchingCore

Continuous batching changes which requests run between model iterations. Learn how a waiting request joins, why an empty slot is not enough, and how to balance throughput against each user’s token latency.

04PagedAttentionPremium

Follow a token position through a block table, watch a cache grow, and trace a shared tail through copy-on-write. Separate memory saved from throughput gained.

05Chunked PrefillPremium

Follow a long prompt through a shared token budget, including its final partial chunk. Learn why chunking can reduce stream pauses without guaranteeing a fixed token gap or a universal throughput gain.

06Speculative DecodingPremium

A draft proposes tokens; the target checks them with a sampling rule that preserves its distribution. Work through the correction after rejection, count accepted drafts separately from emitted tokens, and decide whether the measured round cost earns a speedup.

07Disaggregated Prefill and DecodePremium

Separate prefill and decode to control phase interference, then budget the KV handoff. A short isolated transfer can still overload a shared path; cancellation stops generation before it necessarily releases transfer buffers.

08Prefix Caching and KV ReusePremium

Prefix caching reuses compatible attention state for leading tokens. Calculate shared memory and token reuse, then weigh cache identity, block ownership and routing delay before expecting a latency gain.

09Attention Variants: MHA, GQA, MQA and MLACore

Compare MHA, GQA, MQA and compressed MLA by the state each retains. Derive cache payload in consistent units, distinguish storage capacity from serving throughput, and account for hybrid layers and rank replication.

10Quantization for InferencePremium

Quantization reduces weight, activation or cache precision. Calculate payload plus scales, estimate whole-step speedup, and test task quality before choosing a smaller representation.

11Serving Engines: vLLM, SGLang and TensorRT-LLMPremium

An inference engine coordinates model work and request state. Memory fit and batch throughput do not establish a response deadline; test the actual execution loop before choosing an engine.

12Latency Metrics: TTFT, TPOT and GoodputFree

Read a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

13Multi-LoRA ServingPremium

Multi-LoRA serving applies different adapters to a shared base model. It saves weight memory while adding adapter loading, scheduling and isolation requirements.

14Inference Autoscaling and Cold StartsPremium

Inference autoscaling adds or removes model replicas as demand changes. Queue pressure, latency and startup time help determine how much ready capacity to keep.

15Tokens, Embeddings and LogitsPremium

Trace token IDs, embedding rows and next-token scores. Work out which tensors grow with a batch, when all-position logits waste memory, and why token probabilities do not establish factual confidence.

16Inside a Transformer BlockPremium

Trace one token through attention, a residual connection, normalization and a feed-forward network. See which operations mix token positions, which transform features, and why model layers and GPU execution units are different things.

17Embedding Services and Vector RetrievalPremium

Cosine compares vector direction; raw dot product also depends on length. Index size, ANN recall, evidence relevance and final-answer quality need separate checks.

18Reranking and Retrieval BudgetsPremium

A reranker spends extra computation on retrieved candidates. Count query-document pairs, protect permission boundaries and measure whether better evidence justifies its added queueing and latency.

19Structured Outputs and Constrained DecodingPremium

Constrained decoding rules out tokens that break a grammar. Follow a two-choice example through masking, completion and rollback, then separate valid structure from correct, authorized tool arguments.

20The Serving Optimization WaterfallPremium

Tune an LLM deployment one change at a time against one fixed workload, in an order that follows the binding constraint. A measured run went from 789 to 6,568 output tokens per second, and the two biggest wins were not the knobs people usually reach for first.

21Hybrid State Checkpoints and RollbackPremium

A hybrid model must restore attention history and recurrent state to the same token boundary. Learn why a cache hit can require replay, how rejected draft tokens corrupt an in-place state, and what saved checkpoints cost.

22Voice Turns, Barge-In and Playback BuffersPremium

Voice interruption must stop unwanted generation and invalidate queued playback. Track turn identity, bound audio in milliseconds, and distinguish generated, submitted and actually played output.

23Reasoning Budgets and Answer LatencyPremium

Reasoning tokens consume generation time and retained state before an answer becomes usable. Separate the first generated token from the first answer token, reserve output capacity, and evaluate reasoning limits against task success.

Open Weights & Serving Engines

Serving the 2026 open-weights models: sizing from config.json, latent attention and sparse indexers, vLLM and SGLang arguments, expert parallelism, weight formats, multi-node topologies, benchmarking and capacity planning.

01Open-Weights Models of 2026Free

Compare current Chinese open-weight models by the artifact you can deploy: weight bytes, attention state, supported execution paths, and evaluation conditions. A dated guide to GLM, Kimi, DeepSeek, Qwen, and MiniMax, with worked sizing checks that separate parameter counts from GPU capacity.

02Reading config.json to Size a Model You Have Never RunFree

Read a model configuration as an architecture description, then combine it with tensor metadata and engine allocation rules. Learn which fields determine attention state, why mixed precision needs tensor-level accounting, and how to turn a first estimate into a testable per-GPU capacity plan.

03Multi-Head Latent Attention and Sparse IndexersFree

Latent attention changes what is stored; sparse indexing changes which positions receive expensive attention; recurrent layers summarize history in state. Learn to budget each mechanism separately, including indexer work, cache replication and hybrid prefix snapshots.

04vLLM Server Arguments That MatterFree

Separate executor memory, allocated cache pools and logical occupancy. Map context, sequence and token-batch limits to the constraints they control, then tune a pinned deployment against per-rank memory and latency.

05SGLang Server Arguments That MatterFree

Tune static memory, prefill chunks and admission against measured workload behavior. Separate offline throughput heuristics from interactive queue budgets, and calculate the part of request work that prefix caching can save.

06Expert Parallel and All-to-All BackendsFree

Compare tensor and expert placement without confusing either with sparsity. Derive logical dispatch traffic and redundant-expert memory, then choose backends and balancing from per-rank measurements.

07Weight Formats: FP8 Blocks, MXFP4 and AWQFree

Derive the storage cost of low-precision tensors, including scales, zero points and higher-precision exceptions. Distinguish formats from quantization methods, understand fused dequantization, and test whether a smaller checkpoint actually improves serving capacity without unacceptable quality loss.

08Multi-Node Serving TopologiesFree

Choose what to distribute: complete replicas, tensor operations, layers, experts or request phases. Compare rank-local fit, communication dependencies, workload concurrency and failure domains. A node boundary is not a universal performance boundary, and multiple layouts can be combined.

09Serving Benchmarks That Do Not LieFree

Design reproducible serving experiments with realistic arrivals, cache state and failure accounting. Interpret latency-qualified throughput and conditional per-rank bounds without turning optimistic kernel estimates into production capacity.

10Capacity Planning for Open-Weights FleetsFree

Match forecast traffic to a measured replica operating envelope, then check failure domains, rollout capacity, readiness time and cost. Output tokens alone omit prompt work and live state; total parameter count alone does not determine a replica's throughput.

11Model Onboarding: From Hugging Face to ProductionFree

Turn a model release into a reproducible service: pin the complete artifact, confirm the exact engine path, budget transfer and runtime memory, validate request and response behavior, then benchmark and canary against explicit acceptance criteria.

12Pretraining, Fine-Tuning and DistillationPremium

Understand how a base model becomes a useful application model. Separate training objectives from LoRA and quantization, compare distillation with retrieval, and choose the smallest intervention supported by your evaluation.

Napkin Math & Capacity

The formula sheet as concepts: footprints, FLOPs, intensity, bandwidth bounds, communication volume, cost per token, and how to size a cluster in your head.

01Model Memory FootprintFree

Separate weight payload, growing request state and training allocations. Work through exact units and per-rank budgets, then check the live peak.

02KV Cache SizingFree

Derive cache payload from the stored attention state, then account for mixed sequence lengths, block allocation and per-device limits. Worked examples distinguish exact bytes from deployable capacity.

03Training FLOPs: 6NDFree

Derive the six from one forward and two backward matrix products. Count training tokens, attention and recomputation consistently before turning model work into a schedule.

04GPU-Hours and Time to TrainCore

Convert a stated work budget into elapsed time and GPU-hours. Vary utilization, invert for a deadline, and separate productive time from calendar delays without counting them twice.

05Arithmetic Intensity by OperationCore

Count the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

06Bandwidth-Bound Decode ThroughputPremium

Estimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

07Communication Volume EstimatesPremium

Estimate communication by naming the tensor, collective, group and synchronization schedule. Work through per-rank ring bytes, conditional sharding costs, activation exchanges and routing imbalance before converting traffic into exposed time.

08Cost per Million TokensPremium

Cost per million tokens connects fleet spending to useful output. Keep the time window, token definition and utilization assumptions consistent when comparing deployments.

09TCO: Buy vs RentPremium

Buying and renting GPU capacity have different cost and availability risks. Compare them over a stated lifetime and demand pattern, including idle time and operating costs.

10Capacity Planning and UtilizationPremium

Capacity planning connects demand, useful throughput and acceptable waiting time. Serving and training need different models of bursts, job shape and spare capacity.

11Power and Datacenter ConstraintsPremium

GPU capacity also needs electrical and cooling capacity. Build the facility estimate from complete system loads, rack density and the site’s operating limits.

Networking & Storage

The fabric under the collectives: RDMA, InfiniBand and RoCE, topologies, congestion, GPUDirect, and the storage tiers that feed training and checkpoints.

01NCCL and Collective AlgorithmsFree

NCCL implements GPU collectives using algorithms, protocols and channels over available hardware paths. Learn what each choice changes, how to read bandwidth results, and which evidence separates a communication problem from late application work.

02RDMA, InfiniBand and RoCEv2Core

RDMA lets network adapters transfer data directly between registered memory regions. InfiniBand and RoCE provide different network and operating environments for those transfers.

03Rail-Optimized and Fat-Tree FabricsPremium

Rail-optimized and Clos fabrics organize traffic between GPU nodes. Port counts, placement and oversubscription determine which paths are available to collectives.

04Congestion Control for AI FabricsPremium

Diagnose RoCE congestion using queue growth, ECN feedback and per-priority pause duration. Compare matched traffic patterns before changing routing or flow control.

05GPUDirect RDMA and GPUDirect StoragePremium

GPUDirect enables supported devices to transfer data to or from GPU memory without a host staging buffer. Its benefit depends on topology, software support and the transfer path.

06Topology-Aware CommunicationPremium

Map communication groups onto real GPU and network paths. A four-rank example shows how two tensor-parallel streams share one adapter, why per-rank bandwidth can mislead, and when locality is worth waiting for.

07Parallel Filesystems vs Object StorageCore

Choose storage by access semantics, working-set reuse and failure protection. Trace a shared read/checkpoint budget, distinguish local completion from durable export, and learn when a second storage tier earns its cost.

08Data Loading Pipelines for TrainingPremium

Trace batches from storage to GPU readiness, budget workers and prefetch memory, and resume from the input position saved with training state. An executable example shows how a read-ahead cursor can silently skip samples.

09Checkpoint I/OPremium

Trace checkpoint bytes through staging and storage, calculate when background saves fall behind, and distinguish upload completion from recoverable-state age. Includes an executable queue example and restore budget.

10Debugging a Slow All-ReduceFree

Interpret NCCL algbw and busbw, then isolate a slow collective with controlled tests. Distinguish rank delays, GPU-memory paths and shared-fabric limits without guessing from bandwidth ratios.

11Dataset Lifecycle: Ingest, Shard and RetainFree

Follow a dataset from source records to verified shards, a pinned version and safe retention. Calculate token storage, distinguish caches from authoritative data, and learn why deleting a version does not necessarily free its shards.

12Request Networking: DNS, TCP and TLSPremium

A model can be healthy while requests never reach it. Trace name resolution, connection setup, encryption and HTTP routing; use boundary evidence to separate a network failure from a slow inference engine.

Scheduling & Orchestration

How GPUs become a shared platform: Kubernetes device plugins and DRA, MIG and sharing, gang and topology-aware scheduling, Slurm, quotas, containers and cold starts.

01Kubernetes GPU SchedulingFree

Kubernetes coordinates GPU discovery, scheduling and container access through several components. Understanding their separate roles makes allocation failures easier to diagnose.

02MIG, MPS and Time-SlicingCore

MIG, MPS and time-slicing share a GPU in different ways. Compare memory capacity, execution behavior and isolation before choosing a mechanism.

03Gang Scheduling with Kueue and VolcanoPremium

Gang scheduling treats a distributed job’s required workers as one admission unit. It prevents partial allocations from holding resources while waiting for the rest of the job.

04Topology-Aware SchedulingPremium

Topology-aware scheduling places jobs with the communication network in mind. The right GPU count is not enough when ranks share slow or congested paths.

05Slurm for AI ClustersCore

Slurm allocates resources to batch jobs and launches their tasks. GPU requests, task placement and queue policy work together to determine when a training job runs.

06Slurm vs KubernetesFree

Slurm and Kubernetes provide different foundations for running GPU workloads. Choose the operating model and supporting components that fit the users and jobs.

07Multi-Tenancy, Quotas and Fair SharePremium

Follow three teams through GPU borrowing and reclaim. Distinguish quotas, reservations and fair sharing; see why a returning team may leave another above its nominal share, and why failures or indivisible jobs change what can be promised.

08Ray on KubernetesPremium

Ray schedules Python tasks and actors across workers; KubeRay manages those workers on Kubernetes. The two scheduling layers need compatible resource and placement rules.

09Containers, Images and GPU Cold StartsPremium

GPU cold starts include scheduling, image loading, model loading and engine initialization. Measure each stage before choosing caches, warm pools or supported snapshot mechanisms.

10Spot, Preemption and Capacity StrategiesPremium

Preemptible capacity trades a lower price for interruptions and uncertain availability. Compare useful completed work, billed recovery and capacity wait separately, and restore only a complete checkpoint on a compatible replacement.

11Node Lifecycle: Drain, Upgrade and ReturnFree

Fleet maintenance moves nodes through drain, repair and validation before returning them to service. Deadlines and concurrency limits keep that work within the available capacity.

12Workload Identity and SecretsPremium

A running model service needs permission to read weights and emit telemetry. Separate the workload identity from its credentials, scope access to specific operations, and test expiry, revocation and cross-tenant denial.

13GPU Tenant Isolation BoundariesPremium

GPU allocation, virtual machines and network policies protect different boundaries. Trace each tenant's device, network and storage paths, then test both permitted work and forbidden access before declaring a shared GPU service isolated.

Fleet Reliability & Observability

What breaks in a GPU fleet and how you see it: XID codes, ECC, NVLink, stragglers, thermal, health checks, SLOs, and the incident craft of keeping a training run alive.

01GPU Failure Modes and XID ErrorsFree

XID messages report GPU errors that need interpretation alongside logs and workload behavior. Use the code, recovery guidance and recurrence pattern to choose the next check.

02DCGM and GPU TelemetryCore

DCGM provides GPU telemetry and diagnostics. Combine activity, errors, environmental readings and application progress to understand a fleet’s health.

03ECC, Row Remapping and Memory ErrorsPremium

ECC and row remapping detect or manage some GPU memory faults. Recovery depends on error containment, remapping state and the device’s supported procedure.

04NVLink and Fabric FaultsPremium

GPU link faults can slow collectives or stop distributed progress. Correlate application timing with link counters and controlled path tests to locate the problem.

05Stragglers and HangsPremium

Trace how a late worker reaches the critical path, distinguish longer steps from lower throughput, and separate ordinary jitter from a stalled job. Worked examples show when delay is hidden, how shared jitter changes the math, and what evidence supports a repair.

06Node Health Checks and Burn-InPremium

Health checks look for faults before and during GPU workloads. Burn-in, pre-flight checks and continuous monitoring cover different conditions and need explicit acceptance criteria.

07Training Uptime and Interruption StatisticsFree

Training interruption statistics connect failure frequency to lost progress. Use a stated rate, checkpoint cost and recovery model to estimate how much time a run can lose.

08SLOs for AI SystemsCore

SLOs define the service outcomes users or training teams need. Choose measurable latency, availability and progress targets, then use their error budgets to guide decisions.

09Incident Response for GPU FleetsPremium

GPU incident response restores useful service while preserving enough evidence to diagnose the failure. Mitigation, repair and return-to-service checks have different purposes.

10Thermal, Power and Cooling EventsPremium

Thermal and power limits can reduce GPU performance before a device stops. Clock, throttle and facility evidence help separate these events from other slowdowns.

11Alert Design and On-Call LoadFree

Useful alerts identify a condition that needs a timely response. Connect pages to user impact or imminent risk, and reduce duplicate noise without hiding failures.

12Evaluation Metrics and Data SplitsPremium

Fast predictions can still be wrong. Calculate precision and recall from explicit outcomes, choose thresholds on held-out data, and keep quality evidence separate from serving latency and availability.

13Tracing Inference RequestsPremium

Connect gateway, queue, model execution and streaming delivery without confusing overlapping spans or GPU activity with user-visible latency.

AI Systems Design

The building blocks of the design round: routers, schedulers, queues, caches, autoscalers, multi-tenancy and the numbers that make a design credible.

01Inference Platform ArchitectureFree

An inference platform connects API requests to model replicas. Learn how routing, batching, memory limits and scaling work together to meet a latency target.

02Request Routing and Load Balancing for LLMsPremium

An LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.

03GPU Job Scheduler DesignCore

A GPU scheduler decides which jobs run, where they run and when resources can be reclaimed. Its placement rules must account for whole jobs, topology and tenant policy.

04Training Cluster Design at 10k GPUsPremium

Size a training cluster from the model, data and completion deadline. Then check the network, storage, power and recovery capacity needed to keep those GPUs productive.

05Serverless GPU PlatformsPremium

A serverless GPU platform hides allocation and startup from its users. The design balances cold-start latency, tenant isolation and the cost of idle capacity.

06Multi-Tenant Fine-Tuning ServicePremium

A fine-tuning service turns a tenant’s dataset into an evaluated model or adapter. Its pipeline must protect data while scheduling training and serving the result efficiently.

07Evaluation and Data Pipeline InfrastructurePremium

Data pipelines prepare traceable training inputs; evaluation pipelines compare model behavior. Both need versioned inputs, reproducible execution and useful failure reporting.

08Designing for Latency SLOsCore

A latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.

09Capacity and BackpressurePremium

Backpressure limits accepted work when a service cannot keep up. Bounded queues and clear retry rules prevent overload from spreading through the request path.

10The AI Infra Design Round PlaybookFree

A design interview is a chance to explain requirements, capacity and tradeoffs clearly. Use a flexible structure that leaves time for mechanisms, failures and follow-up questions.

11Control Plane and API Design for GPU PlatformsFree

A control plane records intended work and coordinates its execution. Clear retry, cancellation and listing semantics keep its API useful when requests or workers fail.

12Multi-Region Serving and FailoverFree

Multi-region serving can improve geographic reach and survive a regional outage. It requires enough remaining capacity and a clear plan for state, routing and degraded service.

13Model Metadata and Artifact StoragePremium

Model files and release records have different storage needs. Immutable artifacts and a controlled release pointer help prevent partially uploaded models from reaching users.

14Offline Batch Inference vs Online ServingPremium

Online serving promises each request a latency; a batch job promises a deadline for a whole set and a checked result for every item. That changes what to optimize, when an engine's startup cost dominates, and what delivery means.

15Model Routing and CascadesPremium

Choose a model against measured task quality, latency and cost. Distinguish routing before generation from cascades that pay for a first answer before escalating.

Coding for Infra

The engineering craft infra screens reward: concurrency, backpressure, idempotency, intervals and schedulers, and the patterns behind the practical builds.

01The GPU Credit Scheduler PatternFree

A credit scheduler admits jobs without overspending an account’s balance. Clear state transitions make priority, cancellation and refunds easier to implement correctly.

02Rate-Limiting AlgorithmsCore

Rate limiters control how much work a caller may submit over time. Token buckets, queues and sliding windows allow different burst and waiting behavior.

03Batching Queues and BackpressurePremium

A request batcher groups work to reduce processing overhead. Queue bounds, dispatch timing and failure handling keep that efficiency from becoming uncontrolled waiting.

04Retry, Backoff and IdempotencyCore

Retries can recover transient failures, but they also add load and may repeat side effects. Backoff, retry budgets and idempotency make that behavior explicit.

05Interval Merging and Utilization LogsPremium

Interval merging finds periods of activity and gaps between them. A sweep over start and end events also measures concurrency and utilization.

06Concurrency in Python, Go and C++Premium

Concurrency primitives coordinate overlapping work and shared state. Choose threads, processes or asynchronous tasks from the runtime and the work they must perform.

07Producer-Consumer PipelinesPremium

Producer-consumer pipelines connect stages that run at different rates. Bounded buffers absorb temporary variation, while the slowest sustained stage limits throughput.

08Parsing Kernel Traces and LogsPremium

Trace analysis combines parsing, classification and aggregation. Stream large inputs and define the timing metric before deciding which operations dominate.

09The Practical Coding Screen PlaybookFree

A practical coding interview asks you to build and explain a small system. A clear contract, working core and tests make follow-up changes easier to reason about.

10Cache-Friendly Data StructuresFree

Data layout affects cache misses and communication between CPU cores. Pointer chasing and false sharing explain why equivalent operations can have different memory costs.

11Consistent Hashing and ShardingFree

Consistent hashing limits key movement when a shard set changes. It can preserve cache affinity, but it does not guarantee that cached data is present or that load is balanced.

12Asynchronous GPU Buffer OwnershipPremium

A host function can return while a GPU still uses its pointers. Tie buffer reuse to device completion, including cancellation, cross-stream consumers and foreign-function boundaries.

Ownership & Judgment

The half of the loop most engineers under-prepare: reliability pushback, on-call narratives, working with researchers, and the safety and mission conversations at the labs.

01The Reliability Pushback StoryFree

A reliability pushback story explains the risk you saw, the evidence available, and the cost of delaying or stopping work. Make your responsibility clear and describe what would have changed your decision.

02On-Call Narratives That LandCore

An incident story is easier to assess when it follows the evidence from the first alert to recovery. Explain your decisions, their costs, how you verified the fix, and what the team changed afterward.

03Working with ResearchersPremium

Working with researchers means understanding the experiment’s goal while accounting for shared capacity and reliability. Explain constraints in terms of the research outcome, offer workable alternatives, and agree who owns the decision.

04Migrations and DeprecationsPremium

A migration changes both a system and the workflows that depend on it. Explain why the move is worthwhile, how each rollout stage limits risk, what makes rollback possible, and what users need before the old path can be retired.

05Safety and Mission Rounds at the LabsFree

A mission or values discussion is an opportunity to explain your reasons for joining and how you handle concerns about the work. Use your own views and experience, and connect them to concrete responsibilities such as evaluation and access control.

06Your View on Where AI Infrastructure Is GoingPremium

A useful industry view makes a specific prediction, explains the evidence behind it and identifies what would change your mind. Separate a forecast from an established fact, and keep numerical arguments within their stated assumptions.