GPU Architecture Interview Questions

A grid is a request, a block is a residency unit, a warp is what the scheduler issues. Which of those pins to an SM, why 32 matters, and how a GPU hides a 600 ns memory latency with no branch predictor and a cache that is tiny per thread.

Registers, shared memory, L1, L2, HBM, host memory: sizes, bandwidths and latencies for an H100, derived rather than recited, and the habit of asking 'which level am I hitting' before touching a kernel.

Not 'more cores.' The real answers are a 10x memory system, a 16x matrix datapath, and a design that spends transistors on lanes and registers instead of on making one thread wait less. With the numbers for a two-socket server against one H100.

A tensor core multiplies small matrix tiles in one instruction and is 16x faster than the regular lanes, but only for dense matmul at the right precision, with the right shapes and layouts. What the instruction looks like, what it refuses, and how to tell from a profile whether you are on it.

Peak FLOPS divided by bandwidth is one number per chip, and it decides whether any kernel can ever reach peak. How to compute it, how to compute a kernel's intensity from its bytes and FLOPs, and how to read the answer before writing a line of CUDA.

At batch 1 a decode step reads every weight byte to do two FLOPs with it. The intensity is the batch size, the ridge is about 295, and the KV cache puts a ceiling on how far batching can push you. The full chain for a 70B model on H100.

Every format is a trade between how big a number can be and how finely it is spaced. The bit layouts, the largest and smallest values derived from them, why fp16 needed loss scaling and bf16 did not, and why fp8 comes in two flavors with a scale factor attached.

The H200 has the same compute die as the H100 and costs more per hour. The datasheet gives two ratios, 1.43x bandwidth and 1.76x memory, and those two numbers decide exactly which workloads pay back the premium and which ones lose money on it.

An H100 SXM moves 900 GB/s over NVLink; a PCIe card moves about 64 GB/s per direction over its slot. Whether that 7x to 14x gap costs you anything depends on how many bytes your parallelism strategy sends per step, and the answer is very different for tensor and data parallelism.

A GPU issues one instruction to 32 threads at once. When those threads want different instructions, the hardware runs each path in turn with the others masked off, and the time is the sum of the paths. The derivation of the cost, the metric that exposes it, and the three fixes that work.

Occupancy is a means, and the end is enough bytes in flight to cover memory latency. Little's law gives the number of loads an SM needs outstanding; the register file gives the warps you can afford; the kernels that win on H100 usually run at 25% to 50% occupancy with big tiles and no spills.

HBM stacks DRAM dies on top of each other and wires them to the GPU through a silicon interposer with a 1,024-bit bus per stack. That design sets how much bandwidth and capacity a card can have, why the two scale together, and why they have grown more slowly than FLOPS across three generations.

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

The roofline, the sharding arithmetic and the parallelism trade-offs transfer unchanged, and the numbers are in the same units. What changes is who writes the kernels, how shapes must behave, and the interconnect topology you shard against. The v6e and v7 numbers worked through against an H100.

An N × N grid of multiply-accumulate cells holds a weight tile still and pumps activations through it, one row per cycle, with no register file or instruction fetch in the inner loop. The cycle count, the fill and drain, and the formula that says why a 128-row tile wastes two thirds of the array.

A Trn2 instance carries 16 Trainium2 chips with 1.5 TB of HBM and more dense bf16 FLOPS than an 8-GPU H100 node. Whether the cheaper FLOPS reach your workload depends on the Neuron compiler, the kernels you do not have yet, and an MFU number you must measure rather than assume.

192 GB and 5.3 TB/s per card beat the H100 on both memory numbers at a lower hourly price. Whether that wins in production is a question about achieved bandwidth and ROCm kernel coverage, and there is a short list of numbers to run before you sign the order.

Put the weights in on-chip SRAM and the HBM wall disappears: tens of terabytes per second per chip, and a batch-1 step limited by the pipeline rather than the memory. The price is capacity: a 70B needs hundreds of chips per replica, and the cost per token depends on keeping every one of them busy.

MIG carves an H100 into up to seven hardware slices with their own SMs, memory and bandwidth; MPS lets processes share one SM pool through a single context. One gives fault and performance isolation at a fixed slice size; the other gives flexibility and a shared failure domain. The numbers that decide it.

Four bits give eight magnitudes per sign, and the largest step between them is 50%. Block scaling rescues most of the tensor; a single outlier in a block of sixteen flushes its neighbors to zero. Where the precision goes, which layers fail first, and the measurements that separate acceptable from broken.

512 GPUs at 400 Gb/s each is 205 Tb/s of injection bandwidth a two-tier fabric must carry with no oversubscribed link. The port arithmetic that gives 16 leaves and 8 spines, the rail wiring that keeps NCCL traffic one hop away, and the line where 900 GB/s of NVLink becomes 50 GB/s of InfiniBand.

Without it, every byte a GPU sends over the network crosses PCIe twice and passes through host memory. With it, the NIC reads and writes GPU memory directly. The PCIe arithmetic that shows why a 400G NIC cannot be fed the slow way, the NCCL line that tells you which path you are on, and the checkpoint case.

Four percentages from a profiler summary are enough to place a kernel on the roofline, estimate its arithmetic intensity, and rule out half the optimizations people reach for. The reading, the arithmetic that backs it, and the fix list in the order it pays.

A linear layer at batch 1 runs at a third of a percent of peak and at batch 4,096 near 70%. The curve between has two regions and a sawtooth: the memory-bound slope where intensity equals M, the plateau where the tensor cores are the limit, and the dips where the tile count does not divide the SM count.

An H100 has 50 MB of L2, split into two partitions, with a bandwidth several times HBM's. That is enough to change the roofline for a kernel whose working set fits and to do nothing at all for the ones that stream. The arithmetic of what fits, the residency controls, and the two ways the L2 lies to a profile.

On an HBM part ECC costs almost nothing; on a GDDR part about 6% of capacity and bandwidth. What it buys is the difference between a corrected bit and a silently flipped exponent that turns 1.0 into infinity across a 512-GPU all-reduce. The mechanisms, the signals, and the expected-loss arithmetic.

A two-socket GPU node has two PCIe trees, two memory pools and one inter-socket link that every wrong-sided transfer must cross. A dataloader on the far socket shares that link with seven other GPUs' traffic, and its copies run at half speed. The topology map, the arithmetic and the pinning.

Decode throughput is bytes per step over bandwidth, price is dollars per hour, so tokens per dollar for a 70B falls out of a spec sheet in ten lines. In bf16 the three generations are within 10% per dollar; fp8 pulls the H100 ahead, and fp4 on the B200 doubles it again. The chain, the table and the caveats.

The workload asks for bytes and the chip delivers FLOPs: bytes per FLOP halved from A100 to H100 and held level on B200. A defensible thesis is more bandwidth and capacity per FLOP, the shoreline and power arithmetic for why it has not happened, and the counterargument that software already routes around the wall.

Half the expected bus bandwidth has four plausible causes needing different fixes: a link trained at reduced width, a fabric that never came up, a topology the job did not expect, or a test that measured latency. The sequence of commands that tells them apart, and what each output looks like on a healthy node.