NVIDIA AI Infrastructure Engineer interview questions

Trace a grid into blocks and warps, then distinguish resident work from ready work. Derive an occupancy limit without treating it as a speed target.

Separate storage scope from physical backing, then follow a warp through adjacent and strided loads. Count useful bytes and requested sectors before inferring HBM traffic or speed.

Compare useful parallel work, data movement and reuse. A synthetic CPU/GPU crossover shows why a faster kernel can still lose the end-to-end request.

Follow operand tiles into a matrix-multiply-accumulate instruction. Explain why odd dimensions and low tensor activity do not prove a fallback.

Count two matrix shapes to derive the H100 ridge, throughput ceilings and time floors. Compare an illustrative measurement with its bound before deciding what to optimize.

Use proportional bit layouts and adjacent values around one to distinguish range from precision. Then choose formats from numerical tests and actual kernel support.

Separate H200 bandwidth and capacity gains from measured speedup, then test realistic memory headroom and cost per useful completed workload.

Normalize link directions, count collective bytes, and see why a gradient reduction can outlast its real overlap window.

Compare the same 1,024 items under mixed and grouped lane assignments. Derive issue-work waste, then include regrouping and memory costs before claiming a speedup.

Derive a block-level occupancy limit, then compare the extra residency with spill traffic and lost reuse. Use a matched experiment rather than a target percentage.

Derive HBM bandwidth from interface width and pin rate, then connect it to a model that actually fits. Separate raw interface capacity from measured decode time.

Separate Blackwell chip gains from the 72-GPU domain, follow a token to its expert owners, and test placement instead of dividing model bytes by 72.

See the GPU-instance and compute-instance boundaries, budget a small slice, and distinguish memory capacity controls from bandwidth isolation.

Quantize a sixteen-value block, count the scale bytes, and use paired quality tests to decide which tensors need higher precision.

Derive a 16-leaf, 8-spine fabric from 512 endpoints, trace local and cross-leaf routes, and check what rail placement and oversubscription actually guarantee.

Compare host-staged and direct GPU-to-NIC transfers, count traffic at each boundary, and check transport selection before blaming a slow collective on missing GPUDirect.

Use a profiler summary to form a memory-traffic hypothesis, then check counters, timing and useful work before choosing an optimization.

Calculate the data a kernel could reuse, then test whether it stays in L2. Distinguish capacity, cache-policy hints and measured traffic from a guarantee of residency.

A proposal for more memory bandwidth needs a workload that benefits, a budget for what it costs, and a reason to abandon it. Work through a serving-specific design without assuming all LLM work has the same bottleneck.

Compare the same benchmark on a healthy node, then follow the evidence through NCCL transport choices, device topology and fabric health. Half the measured bandwidth is a symptom, not a unique diagnosis.

Compare GPU memory, decode traffic and measured useful throughput. Work out a capacity ceiling without mistaking it for a performance forecast.

Diagnose failed 800G links using endpoint compatibility, lane modes, inventory and controlled swaps. Learn why speed labels and failure patterns do not prove a cause.

Localize a slow fabric rail with matched tests, path comparisons and counter deltas. Separate an independent-bandwidth sum from collective completion time.

Bring up the supported hardware and software stack one dependency at a time, then prove correctness and recovery at increasing scale.

Size adapter and full-parameter jobs before comparing RTX PRO 6000 and H100. Test memory headroom, MIG packing, software support and real communication costs.

Count one complete rail, then distinguish logical links from cages, optical modules and cable assemblies. Derive power from the same physical inventory.

Diagnose a multi-node regression by comparing host and GPU transfer paths, active registration mechanisms and the tested software stack.

Separate compute, storage and management responsibilities, then decide which need independent physical paths and count their endpoints correctly.

Count endpoints, allocate every switch port once, and compare a common fabric with independent rails before adding a third tier.

Compare 64 or 72 useful GPUs, identify the communication that could improve, and test whether a larger NVLink domain justifies the operational requirements.

Fabric Manager helps make an NVSwitch system ready. Learn the platform-specific roles, failure symptoms and checks that prove the communication path works.

Map stable endpoints to rails, choose supported cables from measured routes, and verify topology before interpreting collective performance.

Confirm the event definition and exposure, compare cohort rates, then test whether the cause follows a module, location or optical path.

Map a non-divisible array onto blocks, guard the final 189 threads, and separate kernel traffic from host transfers without inventing measured timings.

Count warp sectors for contiguous, strided and masked accesses, then use cache and timing evidence to decide whether changing the mapping helps.

Build a mask-aware classifier that counts complete byte intervals, separates duplicate lane demand from distinct bytes, and streams trace summaries.

Derive the bank mapping before and after padding, trace a fixed-tile kernel, and measure whether the saved conflict outweighs other resource costs.

Follow an accumulator across K tiles, count reuse at each memory level, and decide whether a tile improves measured GEMM performance.

Derive resident-block limits and distinguish resident warps from ready work before trading registers for more occupancy.

Compare the mechanisms in FlashAttention versions 1, 2 and 3: IO savings, work partitioning, asynchronous dependencies and FP8 accuracy.

Separate cache activity from DRAM traffic, check the access width and active lanes, and use a controlled layout change to test a profiler hypothesis. A few percentages cannot predict the fixed kernel’s runtime.

Build a reduction with a checked tail, one partial result per block and a final combine. Count the input bytes, then investigate the instructions, synchronization and numerical behavior that a bandwidth estimate leaves out.

Trace how a shared-memory tile turns coalesced input rows into coalesced output rows. Then separate global sector footprints, shared-bank conflicts and measured bandwidth, including the cost of handling edge tiles.

Use a timeline to establish launch overhead, then trace what graph replay changes. Stable buffers, page-table updates, capture-safe operations and batch buckets determine whether the serving engine can use it safely.

Use cuBLAS or cuBLASLt when its supported operation fits. CUTLASS exposes kernel building blocks for custom layouts, fusion and scheduling, but each choice needs a whole-operation benchmark and an ownership plan.

An asynchronous call returns early, but the timeline still shows copies and kernels taking turns. Derive the pipeline's fill and drain costs, then check dependencies, buffers and hardware before expecting overlap.

Watch a low-precision sum stop changing and a small weight update round away. Separate storage precision, accumulation, optimizer state and dynamic range before choosing a mixed-precision training policy.

Derive when operand scales can be applied after a dot product and when they must follow its K blocks. Separate integer zero points, floating-point range, scale granularity and accumulator behavior.

Choose between local candidates, radix selection and sorting from k and the required output. A tested FP32 reference makes ties, signed zero, NaNs and threshold selection explicit.

A few extra live values can change register allocation, residency, generated instructions or spills. Compare compiler evidence with runtime behavior before attributing a 30% slowdown to any one of them.

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Trace a chunk, derive both traffic phases, and state the endpoint model behind bandwidth optimality. A worked example shows why nearly constant bytes do not imply constant completion time.

Trace a legal two-stage schedule and count its idle slots. Separate bubble denominators and explain how 1F1B changes activation lifetimes under explicit assumptions.

Count tensor-parallel messages and compare their payload service time on two explicit paths. Explain why TP8 is a common placement choice, not a fixed ceiling.

Construct a 16384-GPU candidate layout from ownership and batch constraints, with explicit sharding recipes and pipeline idle time.

Separate gradient range from update precision using a small numerical example, and make the training-state recipe explicit.

Overlap starts with a dependency: communication must finish before its consumer needs the result. Estimate the available window, then measure readiness, contention and complete step time for DDP, FSDP or tensor parallelism.

Separate generation, scoring, learning and policy publication. Size the pools from matched workload measurements, and make policy versions explicit before overlapping rollout generation with updates.

GRPO replaces a learned value baseline with comparisons within response groups. Size generation and verification from the actual group workload, then distinguish continuous batching from changes to the training algorithm.

Separate FP8 input quantization from accumulation error. A small representable-value example shows when a shared scale loses small values, while the DeepSeek-V3 report supplies one carefully scoped training recipe.

Compare a complete phase handoff with a tuned combined pool, then test the queueing reversal and qualified completions. Nominal link speed and fleet size alone cannot decide the split.

Filter engine candidates by exact compatibility, then compare correctness, goodput and operational cost on the same workload.

A conversation's cache is leaving GPU memory. Work out whether to save it, reload it later, or recompute, including transfer queues and the chance that nobody returns.

Work two audio timelines: input accumulation before transcription and playback starvation after synthesis. Compare real-time factor, response delay and safe concurrent capacity without mixing their units.

Check the bound’s assumptions, locate the critical path, and quantify which optimization can actually improve one user’s decode latency.

Compare actual encoded tensors, per-rank runtime fit, supported kernels and held-out task quality. Four-bit weight-only kernels can accelerate wider arithmetic, and smaller weights need not reduce the GPU count.

Define whether half the footprint means checkpoint bytes or runtime memory. Choose a supported recipe, measure which tensors it changes, separate calibration from held-out evaluation, and validate quality and serving cost.

Measure exposed launch and dispatch delay, actual graph coverage, capture memory and warmup. Batch size influences the result, but does not alone determine whether graphs improve the service.

Map a checkpoint format to compute capability, then say which part of the speedup survives on each generation. The same file can load on two GPUs and take two different execution paths.

Turn FLOPs and HBM bytes into resource bounds, then compare them with elapsed time. A tighter memory bound suggests an investigation rather than proving the measured bottleneck.

Derive ring all-reduce traffic, match directional bytes to bandwidth, and check memory feasibility before estimating the communication exposed on a training step.

Count weight and cache traffic on a specified B200 configuration, check finite capacity, and separate aggregate throughput from per-user pace. Worked bounds guide a benchmark rather than replace it.

Estimate activation memory from saved tensor shapes and lifetimes. Use a published reference formula carefully, then check checkpointing and parallel ownership against the real peak.

Separate the all-reduce result from NCCL execution choices, and compare latency models without treating them as universal selection rules.

Track actual values across ranks, then count bytes with an explicit buffer definition. Reduction combines values; gathering and all-to-all change ownership.

Choose between supported InfiniBand and RoCE designs using workload tests, congestion behavior and a complete cost model.

Trace registered-memory access and GPU staging, then compare RDMA with sockets using measured CPU and application performance.

Follow corresponding GPU positions through a rail, including the leaf and spine links needed when one rail outgrows a switch.

Calculate queue growth during incast, then distinguish endpoint rate feedback from link-level priority pause and shared-resource contention.

Compare serialized and overlapped storage paths, then test whether direct GPU I/O improves the actual pipeline stage.

Construct concrete TP and DP groups, then verify physical devices and estimate traffic with per-rank rather than whole-model bytes.

Separate NCCL logging, device selection and execution overrides from PyTorch failure handling. Work through a faster collective that makes the job slower, and learn which timeout actually fired.

Derive bandwidth from actual per-rank traffic and readiness deadlines, showing why bucketing cannot erase a capacity shortfall.

Follow expert assignments from logical payload to off-rank and cross-node traffic. Compare dispatch and combine dtypes, receiver skew and local-domain placement.

Distinguish GPU-memory capability, registration and selected transport. Compare direct and staged paths without relying on a half-speed rule or one log substring.

Compare the resource promises of MIG, time-slicing and MPS, then test peak memory and simultaneous demand before packing workloads.

Use actual rank placement and measured job duration to decide whether waiting for locality beats starting immediately.

Separate discovery labels, GPU resource advertisement and validation, and understand the operator as a reconciler with configurable components.

DRA gives the scheduler device claims and attributes to work with. Compare that with extended-resource counts, then check the deployed driver and controller support before promising sharing, partitioning or gang placement.

Interpret XID codes with device and software context, separating immediate recovery from evidence-based root-cause investigation.

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Read link counters with their generation-specific units and traffic exposure. Correlate changes with actual communication performance, then choose monitoring, maintenance or immediate recovery from the supported fault state.

Compare the same collective across the ranks that should participate. Missing records, mismatched arguments, and unfinished operations lead to different next checks.

A reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.

Size both serving pools from workload-matched rates, then check memory, shared transfer paths and surviving capacity. Follow admission and buffer ownership through failure and scaling.

Separate total and active parameters, then size persistent state, approximate compute and token dispatch. Compare supported parallel layouts using per-rank memory, measured communication and recovery.

Stream a trace with an explicit record format, bounded access groups and measured aggregation size. Preserve group identity and report rejected data.

Simulate both phases of ring all-reduce and verify values, preserved inputs and each rank’s payload bytes.

Implement shared and sharded C++ counters, explain relaxed ordering and test final counts without promising unmeasured scaling.

Compare standard-map guarantees with flat-table layouts. Implement bounded probing and choose from semantics and representative measurements.

Implement owner-pop and thief-steal semantics with correct accounting for running tasks that generate children.

Write and verify tiled NumPy matmul, derive its logical footprint and distinguish a cache model from the causes of measured runtime.

Connect an infrastructure problem you want to solve with work you have actually done. Explain what prompted the move now, what transfers from your background and what you still need to learn.

Connect a performance change to its measured mechanism. Compare fair baselines and calculate how a local speedup affects the whole workload.

Build a learning habit around questions, source evidence and practical tests. Explain a real example without claiming work you did not perform.

Move a vendor case forward with controlled comparisons, a faithful reproduction and a safe mitigation. Escalate by impact while evidence improves.

Defend a specific infrastructure opinion with scoped evidence, useful arithmetic and a condition that would change your view.

Turn a design disagreement into a concrete comparison. Use a gang-scheduling example to explain evidence, decision ownership and responsible follow-through.

Build a five-year infrastructure outlook from explicit assumptions, competing explanations and evidence that could change your view. Connect each prediction to a practical engineering decision.

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Follow the running maximum and weighted sum across attention tiles, with executable arithmetic and an honest memory-traffic comparison.

Derive the weight-only shortcut, then add KV traffic and feasible eight-GPU placement. A worked latency floor shows why batch size alone cannot establish the decode bottleneck.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

Trace stable row normalization, handle independent strides and fully masked inputs, then separate fusion traffic from measured speed.

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Draw query-head sharing, count actual stored state and compare payloads with explicit model geometry and binary units.