CUDA & Triton Kernel Interview Questions

Map a non-divisible array onto blocks, guard the final 189 threads, and separate kernel traffic from host transfers without inventing measured timings.

Count warp sectors for contiguous, strided and masked accesses, then use cache and timing evidence to decide whether changing the mapping helps.

Build a mask-aware classifier that counts complete byte intervals, separates duplicate lane demand from distinct bytes, and streams trace summaries.

Derive the bank mapping before and after padding, trace a fixed-tile kernel, and measure whether the saved conflict outweighs other resource costs.

Follow intermediate tensors through separate and fused stages, calculate the traffic and whole-block bounds, and check spills and numerical behavior.

Trace stable row normalization, handle independent strides and fully masked inputs, then separate fusion traffic from measured speed.

Follow an accumulator across K tiles, count reuse at each memory level, and decide whether a tile improves measured GEMM performance.

Derive resident-block limits and distinguish resident warps from ready work before trading registers for more occupancy.

Follow the running maximum and weighted sum across attention tiles, with executable arithmetic and an honest memory-traffic comparison.

Compare the mechanisms in FlashAttention versions 1, 2 and 3: IO savings, work partitioning, asynchronous dependencies and FP8 accuracy.

Start with the operation, supported hardware and performance target. Compare ordinary Triton, lower-level Triton interfaces and CUDA using a measured gap that is large enough to justify maintaining custom code.

Separate cache activity from DRAM traffic, check the access width and active lanes, and use a controlled layout change to test a profiler hypothesis. A few percentages cannot predict the fixed kernel’s runtime.

Build a reduction with a checked tail, one partial result per block and a final combine. Count the input bytes, then investigate the instructions, synchronization and numerical behavior that a bandwidth estimate leaves out.

Trace how a shared-memory tile turns coalesced input rows into coalesced output rows. Then separate global sector footprints, shared-bank conflicts and measured bandwidth, including the cost of handling edge tiles.

Follow graph capture, guards, differentiation and generated kernels. Then distinguish graph breaks from recompilation and measure whether warm execution repays compilation and startup costs.

Use a timeline to establish launch overhead, then trace what graph replay changes. Stable buffers, page-table updates, capture-safe operations and batch buckets determine whether the serving engine can use it safely.

Follow a logical token through the block table into a physical KV page. Then combine split attention states correctly, including empty splits, and count lookup traffic at the layer and head the kernel actually processes.

Use cuBLAS or cuBLASLt when its supported operation fits. CUTLASS exposes kernel building blocks for custom layouts, fusion and scheduling, but each choice needs a whole-operation benchmark and an ownership plan.

Build a numerical reference, count the traffic and investigate a row-wise fused kernel. Separate bandwidth lower bounds from measured performance.

Verify a drafted prefix without biasing the target distribution, then roll back rejected KV state. A six-token prefix exposes the off-by-one: nine committed tokens can have only eight valid KV entries.

An asynchronous call returns early, but the timeline still shows copies and kernels taking turns. Derive the pipeline's fill and drain costs, then check dependencies, buffers and hardware before expecting overlap.

Watch a low-precision sum stop changing and a small weight update round away. Separate storage precision, accumulation, optimizer state and dynamic range before choosing a mixed-precision training policy.

Derive when operand scales can be applied after a dot product and when they must follow its K blocks. Separate integer zero points, floating-point range, scale granularity and accumulator behavior.

Separate per-expert submission, small-grid scheduling and partially filled tiles. A grouped kernel can share work across experts, but routing, packing and output combination still determine the layer’s cost.

Choose between local candidates, radix selection and sorting from k and the required output. A tested FP32 reference makes ties, signed zero, NaNs and threshold selection explicit.

A few extra live values can change register allocation, residency, generated instructions or spills. Compare compiler evidence with runtime behavior before attributing a 30% slowdown to any one of them.

Compute MFU with a stated work model, then follow the training step’s critical path across input, host dispatch, computation and communication. Rank experiments by exposed time instead of assigning universal gains to fixes.

HIP can translate many CUDA interfaces, but it cannot validate the assumptions inside a kernel. Check thread-group widths, masks, shared state and numerical behavior before tuning the MI300X implementation.

Define what the timer includes, give both implementations the same conditions, and keep the sample distribution. A useful benchmark explains the workload and comparison well enough for someone else to reproduce them.

Longer sequences exercise more state updates, different boundaries and sometimes different kernel paths. Use a two-tile example to check online softmax, then test masking, indexing and precision against a stable reference.

Derive row-major offsets, broadcast the bias correctly, and compare a lane-by-lane CPU reference with a vector expression. Explain when shared memory adds no reuse.

A canceled request returns its buffers to a pool while device work is still outstanding. Trace ownership through two streams and design a safe reclamation path.

Separate compilation, module loading and execution before judging a faster kernel. Define the artifact and startup evidence needed to release a compiler upgrade.

Derive the ideal gain from filling a partial wave, then account for partial outputs, reduction order, workspace and the epilogue before selecting split-K.

Trace local counts into globally disjoint output ranges. Find the missing block carry and explain why a spin-wait barrier can deadlock the grid.

Review the input contract, adversarial cases, evaluator independence and compiled baseline before trusting a generated GPU kernel or its reported speedup.