AI Infra Interviews logo

Model GPU work mapping and parallel scans

Run CPU models for GEMM scheduling, tensor offsets, prefix scans and softmax state merging. Inspect intermediate values and test the assumptions before writing a GPU kernel.

PremiumAdvanced

AI Infra Interviews editorial · Updated

Can you learn GPU work partitioning and prefix scans without a GPU?

Yes. CPU models can verify tile counts, scheduling assumptions, tensor offsets and the block carries that make output ranges disjoint. They cannot establish GPU speed, memory ordering or device-wide progress. This project uses small synthetic arrays, explicit intermediate values and negative controls so you can separate index correctness from the hardware measurements that follow.

Read the primary sources
What you build
Executable work-mapping models, a synthetic evidence file and tested ownership calculations.
Environment
Python 3.10+ · CPU only
Plan your session
60–90 minutes. A planning estimate, not measured runtime.
Before you begin
Read the kernel course’s work-mapping module. Be comfortable running Python scripts and following array indices.
13 CPU tests executed. CPU arithmetic and negative controls executed on Linux/Python 3.12. Synthetic wave plans are not GPU speed measurements; device synchronization and fixed-width overflow need separate validation.
  1. 01Predict the work

    Calculate tiles, slots and offsets with explicit assumptions.

  2. 02Check ownership

    Scan local counts, add block carries and reject overlapping ranges.

  3. 03State the limits

    Separate calculated consequences from device behavior that still needs measurement.

Follow the changing state through the project. Each boundary has a check in the walkthrough.

Continue with Premium

Premium includes this guide’s complete walkthrough and downloadable project files. The technical overview, prerequisites and verification notes above are public.

View Premium access

Start with Train a tiny language model from scratch with PyTorch (Free to read).

Primary sources

Checked . Source review and execution checks are described separately above.