Easy AI infra interview questions

Trace a grid into blocks and warps, then distinguish resident work from ready work. Derive an occupancy limit without treating it as a speed target.

Separate storage scope from physical backing, then follow a warp through adjacent and strided loads. Count useful bytes and requested sectors before inferring HBM traffic or speed.

Compare useful parallel work, data movement and reuse. A synthetic CPU/GPU crossover shows why a faster kernel can still lose the end-to-end request.

Follow operand tiles into a matrix-multiply-accumulate instruction. Explain why odd dimensions and low tensor activity do not prove a fallback.

Map a non-divisible array onto blocks, guard the final 189 threads, and separate kernel traffic from host transfers without inventing measured timings.

Count warp sectors for contiguous, strided and masked accesses, then use cache and timing evidence to decide whether changing the mapping helps.

Follow intermediate tensors through separate and fused stages, calculate the traffic and whole-block bounds, and check spills and numerical behavior.

Count gradient traffic from a named dtype, separate initialization and buffer broadcasts, and measure the reduction tail instead of assuming it is hidden.

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Compute MFU from a declared global token count and time interval. Check the accounting before judging the percentage or choosing an optimization.

Track which state remains replicated at each ZeRO stage. Compute a declared 70B memory recipe, then add live gathers, activations and buffers before deciding fit.

Derive the weighted gradient, test a short micro-batch, and separate activation savings from total memory and communication. Matching the optimizer step is only part of matching a larger batch.

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

One request finishes, another keeps writing, and a third waits outside. Follow the same workload through two schedules to see what continuous batching improves, and what it cannot promise.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Compare raw weight sizes at 16, 8 and 4 bits, then budget KV cache and runtime memory for Llama 3.1 70B. Follow the worked per-GPU example.

Calculate Llama 3.1 70B KV cache memory at 128k context using eight KV heads. Worked BF16 and FP8 payloads, GB versus GiB, and per-GPU capacity checks.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

Work from training FLOPs to fleet throughput, days and GPU-hours. Then see how utilization and downtime change the estimate.

Check model placement first, then derive a TP8 traffic floor and add context. Separate a single stream’s pace from an ideal bandwidth ratio.

Separate the all-reduce result from NCCL execution choices, and compare latency models without treating them as universal selection rules.

Track actual values across ranks, then count bytes with an explicit buffer definition. Reduction combines values; gathering and all-to-all change ownership.

Trace registered-memory access and GPU staging, then compare RDMA with sockets using measured CPU and application performance.

Follow a four-GPU request from advertised capacity to concrete devices, and see why free GPUs across nodes cannot satisfy one pod.

Compare the resource promises of MIG, time-slicing and MPS, then test peak memory and simultaneous demand before packing workloads.

See how partial allocation can strand two training jobs. Separate group admission, real placement, readiness and recovery instead of assuming atomic pod startup.

Separate discovery labels, GPU resource advertisement and validation, and understand the operator as a reconciler with configurable components.

Use the Llama 3 interruption record without turning job interruptions into device lifetime, predictable failures or a universal spare percentage.

Interpret XID codes with device and software context, separating immediate recovery from evidence-based root-cause investigation.

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Use a flexible 45-minute plan to connect requirements, a request path, capacity arithmetic, selected deep dives and a clear closing decision.

Build a small decimal billing calculator with an explicit rounding policy, exact durations and tests for invalid records and credits.

Two parameters, one lazy refill and one lock. What capacity and rate each control and why conflating them is the usual bug, the clock choice that avoids a whole class of failure, and the retry-after value that turns a rejection into something a client can act on.

The union is a sort and a scan. The intersection is the same sweep with a counter, and it is the one that goes wrong, because a single GPU reporting overlapping intervals will be counted twice and produce an answer that looks plausible. Both implementations, the trap, and the tests that catch it.

Implement bounded retries with full jitter, a shared credit budget and a cooperative deadline. Test repeatability and explain what a timeout leaves uncertain.

Implement stable temperature, top-k and top-p sampling with explicit tie rules, masks, invalid inputs and probability tests.

Explain a reliability concern with evidence, a specific proposal and an honest outcome. Show how you weighed the cost of delaying a launch and what changed afterward.

Tell an incident through the decisions you made, the evidence available at the time and the cost of recovery. Keep your role clear and distinguish mitigation from finding the root cause.

Connect an infrastructure problem you want to solve with work you have actually done. Explain what prompted the move now, what transfers from your background and what you still need to learn.

Connect a performance change to its measured mechanism. Compare fair baselines and calculate how a local speedup affects the whole workload.

Build a learning habit around questions, source evidence and practical tests. Explain a real example without claiming work you did not perform.