Google Cloud and TPU GPU & Accelerator Architecture interview questions

A grid is a request, a block is a residency unit, a warp is what the scheduler issues. Which of those pins to an SM, why 32 matters, and how a GPU hides a 600 ns memory latency with no branch predictor and a cache that is tiny per thread.

Not 'more cores.' The real answers are a 10x memory system, a 16x matrix datapath, and a design that spends transistors on lanes and registers instead of on making one thread wait less. With the numbers for a two-socket server against one H100.

The roofline, the sharding arithmetic and the parallelism trade-offs transfer unchanged, and the numbers are in the same units. What changes is who writes the kernels, how shapes must behave, and the interconnect topology you shard against. The v6e and v7 numbers worked through against an H100.

An N × N grid of multiply-accumulate cells holds a weight tile still and pumps activations through it, one row per cycle, with no register file or instruction fetch in the inner loop. The cycle count, the fill and drain, and the formula that says why a 128-row tile wastes two thirds of the array.

Registers, shared memory, L1, L2, HBM, host memory: sizes, bandwidths and latencies for an H100, derived rather than recited, and the habit of asking 'which level am I hitting' before touching a kernel.

A tensor core multiplies small matrix tiles in one instruction and is 16x faster than the regular lanes, but only for dense matmul at the right precision, with the right shapes and layouts. What the instruction looks like, what it refuses, and how to tell from a profile whether you are on it.

Peak FLOPS divided by bandwidth is one number per chip, and it decides whether any kernel can ever reach peak. How to compute it, how to compute a kernel's intensity from its bytes and FLOPs, and how to read the answer before writing a line of CUDA.

At batch 1 a decode step reads every weight byte to do two FLOPs with it. The intensity is the batch size, the ridge is about 295, and the KV cache puts a ceiling on how far batching can push you. The full chain for a 70B model on H100.

Every format is a trade between how big a number can be and how finely it is spaced. The bit layouts, the largest and smallest values derived from them, why fp16 needed loss scaling and bf16 did not, and why fp8 comes in two flavors with a scale factor attached.

The H200 has the same compute die as the H100 and costs more per hour. The datasheet gives two ratios, 1.43x bandwidth and 1.76x memory, and those two numbers decide exactly which workloads pay back the premium and which ones lose money on it.

An H100 SXM moves 900 GB/s over NVLink; a PCIe card moves about 64 GB/s per direction over its slot. Whether that 7x to 14x gap costs you anything depends on how many bytes your parallelism strategy sends per step, and the answer is very different for tensor and data parallelism.

A GPU issues one instruction to 32 threads at once. When those threads want different instructions, the hardware runs each path in turn with the others masked off, and the time is the sum of the paths. The derivation of the cost, the metric that exposes it, and the three fixes that work.

Occupancy is a means, and the end is enough bytes in flight to cover memory latency. Little's law gives the number of loads an SM needs outstanding; the register file gives the warps you can afford; the kernels that win on H100 usually run at 25% to 50% occupancy with big tiles and no spills.

HBM stacks DRAM dies on top of each other and wires them to the GPU through a silicon interposer with a 1,024-bit bus per stack. That design sets how much bandwidth and capacity a card can have, why the two scale together, and why they have grown more slowly than FLOPS across three generations.

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

A Trn2 instance carries 16 Trainium2 chips with 1.5 TB of HBM and more dense bf16 FLOPS than an 8-GPU H100 node. Whether the cheaper FLOPS reach your workload depends on the Neuron compiler, the kernels you do not have yet, and an MFU number you must measure rather than assume.

192 GB and 5.3 TB/s per card beat the H100 on both memory numbers at a lower hourly price. Whether that wins in production is a question about achieved bandwidth and ROCm kernel coverage, and there is a short list of numbers to run before you sign the order.

Put the weights in on-chip SRAM and the HBM wall disappears: tens of terabytes per second per chip, and a batch-1 step limited by the pipeline rather than the memory. The price is capacity: a 70B needs hundreds of chips per replica, and the cost per token depends on keeping every one of them busy.

MIG carves an H100 into up to seven hardware slices with their own SMs, memory and bandwidth; MPS lets processes share one SM pool through a single context. One gives fault and performance isolation at a fixed slice size; the other gives flexibility and a shared failure domain. The numbers that decide it.