The cluster passed every acceptance test and production throughput is 20 percent below plan. Where do you look?

Acceptance measures the hardware and production measures the whole system, so the gap lives in everything acceptance did not cover. Four candidate layers, the order to eliminate them, and the two that account for most of these cases.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🖧 Hardware & Cluster Build-Out

Burn-In and Acceptance TestingNew hardware fails early or it fails late, and burn-in exists to move the early failures before the cluster is handed over rather than after. A proper acceptance test runs every layer under sustained load for days, compares every node against its siblings rather than against a specification, and produces a signed number the buyer and the vendor both agree on. The comparison is the important part: identical hardware running identical work should produce identical numbers, and the outliers are the finding.

Foundational

🖧 Hardware & Cluster Build-Out

Direct-to-Chip Liquid Cooling and CDUsAbove roughly 40 kW a rack cannot be cooled by air in any practical hall, which is why every dense GPU deployment now runs liquid to the chip. A cold plate sits on each GPU, a coolant distribution unit isolates the clean rack loop from facility water, and the facility side runs warm, typically 30 to 40 degrees supply, because warm water is cheaper to make. The design numbers are flow rate and temperature rise, and both fall out of one equation that every operator should be able to do from memory.

Foundational

🧩 GPU & Accelerator Architecture

Roofline ModelThe roofline plots a kernel's attainable throughput against its arithmetic intensity, FLOPs per byte moved from memory. Below the ridge point (peak FLOPS divided by memory bandwidth, about 295 on an H100 in bf16) a kernel is memory-bound and no amount of clever code reaches the peak; above it, compute is the limit. One picture explains why decode runs at under 1% of peak and why fusion and batching are the two levers that move it.

Foundational

💻 Coding for Infra

Cache-Friendly Data StructuresA cache line is 64 bytes and it is the unit of coherence, so where data sits decides how fast code runs more often than which algorithm it uses. Two consequences dominate infrastructure code: a lookup that chases a pointer pays two dependent memory stalls instead of one, and two threads updating adjacent variables contend for a line they do not logically share. Both are layout problems with layout fixes.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackRoll out a driver and firmware upgrade across a live 2,048-GPU fleet without losing a training run.Next in this trackA new GPU generation is arriving. Plan the migration of a live 2,048-GPU cluster.Next in this trackA vendor claims their accelerator beats an H100 at half the price. How do you evaluate that?

FEDITOR'S NOTE

Scored on separating what acceptance tested from what production exercises, on eliminating layers in cost order, and on naming thermal steady state and software configuration as the usual causes.

DISCUSSION · 0

No comments yet — be the first to share your approach.