AMD CUDA, Triton & Kernel Engineering interview questions

HIP can translate many CUDA interfaces, but it cannot validate the assumptions inside a kernel. Check thread-group widths, masks, shared state and numerical behavior before tuning the MI300X implementation.

Follow the running maximum and weighted sum across attention tiles, with executable arithmetic and an honest memory-traffic comparison.

Derive resident-block limits and distinguish resident warps from ready work before trading registers for more occupancy.

Trace stable row normalization, handle independent strides and fully masked inputs, then separate fusion traffic from measured speed.

Map a non-divisible array onto blocks, guard the final 189 threads, and separate kernel traffic from host transfers without inventing measured timings.

Follow intermediate tensors through separate and fused stages, calculate the traffic and whole-block bounds, and check spills and numerical behavior.

Compare the mechanisms in FlashAttention versions 1, 2 and 3: IO savings, work partitioning, asynchronous dependencies and FP8 accuracy.

Count warp sectors for contiguous, strided and masked accesses, then use cache and timing evidence to decide whether changing the mapping helps.

Follow an accumulator across K tiles, count reuse at each memory level, and decide whether a tile improves measured GEMM performance.

Separate cache activity from DRAM traffic, check the access width and active lanes, and use a controlled layout change to test a profiler hypothesis. A few percentages cannot predict the fixed kernel’s runtime.

Build a mask-aware classifier that counts complete byte intervals, separates duplicate lane demand from distinct bytes, and streams trace summaries.

Derive the bank mapping before and after padding, trace a fixed-tile kernel, and measure whether the saved conflict outweighs other resource costs.

Build a numerical reference, count the traffic and investigate a row-wise fused kernel. Separate bandwidth lower bounds from measured performance.

Compute MFU with a stated work model, then follow the training step’s critical path across input, host dispatch, computation and communication. Rank experiments by exposed time instead of assigning universal gains to fixes.

Verify a drafted prefix without biasing the target distribution, then roll back rejected KV state. A six-token prefix exposes the off-by-one: nine committed tokens can have only eight valid KV entries.

Start with the operation, supported hardware and performance target. Compare ordinary Triton, lower-level Triton interfaces and CUDA using a measured gap that is large enough to justify maintaining custom code.