Lambda AI Infrastructure Engineer interview questions

Separate H200 bandwidth and capacity gains from measured speedup, then test realistic memory headroom and cost per useful completed workload.

Normalize link directions, count collective bytes, and see why a gradient reduction can outlast its real overlap window.

Derive a 16-leaf, 8-spine fabric from 512 endpoints, trace local and cross-leaf routes, and check what rail placement and oversubscription actually guarantee.

Trace a changed BF16 bit, distinguish corrected and uncorrectable errors, and calculate recovery cost without assuming every device has the same ECC overhead.

Follow a host buffer across a NUMA boundary, calculate the cost with declared bandwidths, and align CPU workers and memory with the GPU that consumes the data.

Specify a cluster from workload and facility constraints, then derive node counts, network ports, storage bursts and acceptance tests.

Diagnose failed 800G links using endpoint compatibility, lane modes, inventory and controlled swaps. Learn why speed labels and failure patterns do not prove a cause.

Build a cluster power budget with clear equipment boundaries, facility overhead and failover requirements. Distinguish energy use from reserved electrical capacity.

Bring up the supported hardware and software stack one dependency at a time, then prove correctness and recovery at increasing scale.

Size adapter and full-parameter jobs before comparing RTX PRO 6000 and H100. Test memory headroom, MIG packing, software support and real communication costs.

Investigate rack power events at the correct electrical boundary and time scale. Test load reductions without assuming GPU caps control every watt or transient.

Diagnose a multi-node regression by comparing host and GPU transfer paths, active registration mechanisms and the tested software stack.

Design a management path that survives specified production failures. Count all endpoint classes, restrict device authority and test the complete recovery route.

Compare cooling methods from the supported hardware and heat path. Calculate airflow or liquid flow, then verify capacity, residual air load and failure response.

Estimate replacement demand over replenishment time, then size for shortage risk and failure correlation. Distinguish shelf parts from runnable spare nodes.

Separate compute, storage and management responsibilities, then decide which need independent physical paths and count their endpoints correctly.

Compare ownership, colocation and rental using the same useful work, full annual costs and realistic demand. Work a break-even from explicit assumptions.

Fabric Manager helps make an NVSwitch system ready. Learn the platform-specific roles, failure symptoms and checks that prove the communication path works.

Choose a supported tenant boundary, secure the surrounding services, and verify cleanup before reassignment. Calculate fragmentation from actual requests.

Protect running work, validate a supported upgrade bundle, and expand through measured cohorts with a tested recovery path.

Test the vendor claim on your workload, including quality, latency, scaling and operating costs. A lower peak or lower price alone does not settle it.

Keep the full failure history and investigate what changed. One passing retry is evidence, but it is not a repair or an acceptance policy.

Define who owns each facility and hardware action, model restoration time, and verify power, cooling, telemetry and growth commitments before deployment.

Collect stable device identities, versions and freshness, then compare them with approved configurations and retain the history for investigations.

Compare the old fleet's useful value with operating costs and resale, verify its health, and complete data handling before reassignment or transfer.

An asynchronous call returns early, but the timeline still shows copies and kernels taking turns. Derive the pipeline's fill and drain costs, then check dependencies, buffers and hardware before expecting overlap.

Compare ownership with the actual rental contract, using explicit cash flows, resale assumptions and the hours that can really be avoided.

Calculate the fleet, duration and rental budget for a 7B training run. Check the deadline, actual sharding scheme and paid reservation period before placing the order.

A bandwidth regression narrows the search only after the experiment is defined. Work through rank-count normalization, controlled path tests and the difference between collective time and training progress.

Count endpoint, leaf, aggregation and core ports for an explicit 4,096-GPU design. Compare 1:1 and 2:1 without confusing switch ports, links or optics.

Separate host-buffer, GPU-buffer and collective tests, with explicit message-size models and bandwidth metric definitions.

Distinguish GPU-memory capability, registration and selected transport. Compare direct and staged paths without relying on a half-speed rule or one log substring.

Balance durable checkpoint cost against replay and restart loss, with explicit assumptions and a warning sign when the approximation breaks down.

Compare the cost of committed capacity with the demand each additional GPU would actually serve. Keep billing discounts separate from capacity guarantees, and account for bursts without counting the same work twice.

Use a short admission check for recent changes and a deeper acceptance suite for new or repaired nodes. Set thresholds from matched healthy baselines, and make quarantine and return-to-service decisions explicit.

Follow a GPU request from node inventory through allocation, device confinement and task launch. A single-node, eight-task example shows how resource shape and per-task device numbering fit together.

Treat the upgrade as a tested change to a compatibility set. Drain safely, canary representative workloads, size waves from usable spare capacity and rehearse restoration of the previous working configuration.

Use the Llama 3 interruption record without turning job interruptions into device lifetime, predictable failures or a universal spare percentage.

Interpret XID codes with device and software context, separating immediate recovery from evidence-based root-cause investigation.

Handle an inaccessible GPU with bounded recovery and preserved evidence, without treating a reboot or repeat count as proof of root cause.

Find the critical-path delay using local phase timing and controlled comparisons, rather than diagnosing a node from utilization or clock ratios.

Build an acceptance suite around the node’s intended workload and supported hardware limits. Combine configuration, correctness, communication and sustained-load evidence, then record a baseline and an explicit return-to-service decision.

Separate removing a suspect GPU from production from proving it qualifies for replacement. Use vendor recovery requirements, device history, targeted tests, and the cost of recurrence.

Follow GPU ownership from reservation through verified delivery and cleanup, with stale-operation fencing, recovery and auditable metering.

Evaluate a proposed 2,000-GPU purchase using compatible capacity, dated demand and complete alternatives. Make any gap left by reclaim visible.

Move a vendor case forward with controlled comparisons, a faithful reproduction and a safe mitigation. Escalate by impact while evidence improves.

Explain a career move using the actual role, team and your priorities. Test assumptions about ownership, hardware access and support.

Compare raw weight sizes at 16, 8 and 4 bits, then budget KV cache and runtime memory for Llama 3.1 70B. Follow the worked per-GPU example.

Separate quota, borrowing, reclaim latency and fairness, with a demand-conserving example of when pooling improves utilization.

Work from training FLOPs to fleet throughput, days and GPU-hours. Then see how utilization and downtime change the estimate.

Follow a four-GPU request from advertised capacity to concrete devices, and see why free GPUs across nodes cannot satisfy one pod.

DRA gives the scheduler device claims and attributes to work with. Compare that with extended-resource counts, then check the deployed driver and controller support before promising sharing, partitioning or gang placement.

Solve for useful throughput, round to a supported parallel layout, and account for calendar time once. The right GPU count need not be a power of two.

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Separate frozen weights from trainable state, count rank-16 adapters from matrix shapes, and check peak memory before choosing a fine-tuning layout.