AI Infra Interviews logo
GPU & Accelerator Architecture / 15
hardNewGoogleGroqCerebras

Explain how a systolic array computes a matrix multiply. Why is it efficient, and why does it want big square tiles?

An N × N grid of multiply-accumulate cells holds a weight tile still and pumps activations through it, one row per cycle, with no register file or instruction fetch in the inner loop. The cycle count, the fill and drain, and the formula that says why a 128-row tile wastes two thirds of the array.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

An N × N grid of multiply-accumulate cells holds a weight tile still and pumps activations through it, one row per cycle, with no register file or instruction fetch in the inner loop. The cycle count, the fill and drain, and the formula that says why a 128-row tile wastes two thirds of the array.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced
🧩 GPU & Accelerator Architecture🔒 Premium
TPU Architecture and Systolic ArraysA TPU is a matrix unit first and a processor second: a systolic array that streams activations through a grid of multiply-accumulate cells holding stationary weights, fed by a compiler rather than a warp scheduler, and scaled out over a dedicated inter-chip interconnect into pods of thousands. The roofline thinking transfers from GPUs unchanged; the workflow does not, which is what the Google loop tests.
Advanced
Kernels & Compilers🔒 Premium
Tiled Matrix MultiplicationA matrix multiply has enough reuse to be compute-bound, but only if the kernel captures that reuse in shared memory and registers instead of re-reading HBM. Tiling is how: a block owns an output tile, streams K-slices of A and B through shared memory, and each thread accumulates a small register tile. It is the live-coding exercise that separates people who know the roofline from people who have climbed it.
Core
🧩 GPU & Accelerator ArchitectureSign in
Tensor Cores and Matrix UnitsTensor cores are fixed-function units that compute a small matrix multiply-accumulate per instruction, and they are where almost all of a modern GPU's FLOPS live: 989 dense bf16 TFLOPS on an H100 against about 67 from the general-purpose lanes. Only dense, well-shaped matrix multiplication at a supported precision can use them, which is why GEMMs reach peak and nothing else does, and why precision choices are throughput choices.
Advanced
🧩 GPU & Accelerator Architecture🔒 Premium
Cerebras, Groq and Dataflow AcceleratorsCerebras and Groq attack the same wall from the same direction: keep the working set in on-chip SRAM, which is tens of times faster than HBM, so that memory-bound decode stops being memory-bound. Cerebras does it with a wafer-scale engine holding 44 GB of SRAM; Groq with a deterministic, compiler-scheduled dataflow chip holding a few hundred megabytes per part and many parts per model. Both win single-stream decode by large factors and pay in capacity, cost per token at high batch, and flexibility. Knowing where each fits is the interview question.
UP NEXT ON YOUR JOURNEY
FEDITOR'S NOTE

Scored on a correct data-flow picture (stationary weights, activations shifting one cell per cycle, partial sums accumulating down the column), the cycle-count derivation, and the efficiency formula as a function of tile size.

DISCUSSION · 0

No comments yet — be the first to share your approach.