Meta AI Infrastructure Engineer interview questions

Use proportional bit layouts and adjacent values around one to distinguish range from precision. Then choose formats from numerical tests and actual kernel support.

Specify a cluster from workload and facility constraints, then derive node counts, network ports, storage bursts and acceptance tests.

Compare rack-level power, cooling, structural and connectivity requirements with the offered site, then evaluate quoted deployment alternatives.

Diagnose failed 800G links using endpoint compatibility, lane modes, inventory and controlled swaps. Learn why speed labels and failure patterns do not prove a cause.

Derive cooling flow for a 4.8 MW hall, separate liquid and air loads, and test CDU capacity and redundancy at the actual design conditions.

Localize a slow fabric rail with matched tests, path comparisons and counter deltas. Separate an independent-bandwidth sum from collective completion time.

Bring up the supported hardware and software stack one dependency at a time, then prove correctness and recovery at increasing scale.

Define acceptance with component and system tests, absolute targets, peer comparisons and documented retests. Interpret burn-in exposure without treating it as a reliability guarantee.

Count one complete rail, then distinguish logical links from cages, optical modules and cable assemblies. Derive power from the same physical inventory.

Diagnose a multi-node regression by comparing host and GPU transfer paths, active registration mechanisms and the tested software stack.

Build a storage budget that includes retained checkpoints, the next save, overlapping datasets and failure protection. Then check whether the same tier can meet write and restore deadlines.

Build a deployment schedule from dependencies and committed lead times. Calculate float and price the delay only when an action moves usable capacity earlier.

Estimate replacement demand over replenishment time, then size for shortage risk and failure correlation. Distinguish shelf parts from runnable spare nodes.

Investigate widespread acceptance failures using matched controls, valid thresholds and cohort denominators. Preserve hardware and shared causes as testable hypotheses.

Count endpoints, allocate every switch port once, and compare a common fabric with independent rails before adding a third tier.

Compare 64 or 72 useful GPUs, identify the communication that could improve, and test whether a larger NVLink domain justifies the operational requirements.

Budget the full megawatt, fit whole servers, and compare useful training throughput under measured power limits. A worked cap example shows when more GPUs help.

Fabric Manager helps make an NVSwitch system ready. Learn the platform-specific roles, failure symptoms and checks that prove the communication path works.

Map stable endpoints to rails, choose supported cables from measured routes, and verify topology before interpreting collective performance.

Reproduce the baseline, compare equivalent work, and trace the missing time. Acceptance, production and the original plan may differ in more than one way.

Protect running work, validate a supported upgrade bundle, and expand through measured cohorts with a tested recovery path.

Plan a GPU-generation migration around usable workload capacity, site readiness, compatibility and a tested rollback path.

Test the vendor claim on your workload, including quality, latency, scaling and operating costs. A lower peak or lower price alone does not settle it.

Gate admission with fresh, platform-aware evidence. Use absolute and peer baselines, budget test overhead and keep automated cordons under control.

Keep the full failure history and investigate what changed. One passing retry is evidence, but it is not a repair or an acceptance policy.

Survey real facility headroom, phase work around shared dependencies, and compare retrofit with relocation using useful capacity and total transition cost.

Confirm the event definition and exposure, compare cohort rates, then test whether the cause follows a module, location or optical path.

Collect stable device identities, versions and freshness, then compare them with approved configurations and retain the history for investigations.

Compare complete fabric proposals on workload performance, support and total cost. Translate any measured regression into application impact before pricing the discount.

Compare the old fleet's useful value with operating costs and resale, verify its health, and complete data handling before reassignment or transfer.

Diagnose a rack outage through independent evidence, preserve safety interlocks, and restore capacity only after its failed paths pass acceptance checks.

Follow graph capture, guards, differentiation and generated kernels. Then distinguish graph breaks from recompilation and measure whether warm execution repays compilation and startup costs.

Watch a low-precision sum stop changing and a small weight update round away. Separate storage precision, accumulation, optimizer state and dynamic range before choosing a mixed-precision training policy.

Compute MFU with a stated work model, then follow the training step’s critical path across input, host dispatch, computation and communication. Rank experiments by exposed time instead of assigning universal gains to fixes.

Count gradient traffic from a named dtype, separate initialization and buffer broadcasts, and measure the reduction tail instead of assuming it is hidden.

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Compute MFU from a declared global token count and time interval. Check the accounting before judging the percentage or choosing an optimization.

Track which state remains replicated at each ZeRO stage. Compute a declared 70B memory recipe, then add live gathers, activations and buffers before deciding fit.

Compare FSDP2 and ZeRO-3 through one parameter group’s lifetime, a scoped traffic model and a save-resume experiment, rather than a universal speed ranking.

Trace a legal two-stage schedule and count its idle slots. Separate bubble denominators and explain how 1F1B changes activation lifetimes under explicit assumptions.

Derive the weighted gradient, test a short micro-batch, and separate activation savings from total memory and communication. Matching the optimizer step is only part of matching a larger batch.

Follow saved and recomputed values through backward. Compare complete update time and peak memory instead of promising a fixed memory saving or slowdown.

Trace a causal query across local and remote KV blocks. Separate sequence and context sharding, count one concrete tensor, and avoid dividing memory twice.

Follow one token to two expert ranks and back, then count logical assignments separately from physical network transfers.

Construct a 16384-GPU candidate layout from ownership and batch constraints, with explicit sharding recipes and pipeline idle time.

Separate gradient range from update precision using a small numerical example, and make the training-state recipe explicit.

Balance checkpoint pauses against lost work using job-level interruption data, while accounting for asynchronous durability lag.

Treat recovery as a new worker generation, then compare spare replacement with a legal shrink and its batch consequences.

Separate a longer step from lower throughput, trace the dependency that changed, and distinguish input, workload, device and path causes. Then calculate whether a repair recovers more time than it costs.

Read a watchdog timeout without blaming the reporting rank. Compare group-scoped traces, separate missing evidence from mismatched calls, and distinguish CPU submission from GPU completion.

Overlap starts with a dependency: communication must finish before its consumer needs the result. Estimate the available window, then measure readiness, contention and complete step time for DDP, FSDP or tensor parallelism.

Describe logical tensors independently of writer ranks, publish only a complete checkpoint, and reconstruct the target layout from stored chunks. Asynchronous saving reduces blocking but still needs a consistent snapshot and a memory budget.

Turn the daily loss into recovery and replay costs, then identify which part is measured and which is assumed. Compare checkpoint cadence, recovery work and failure prevention without promising one universal floor.

Build a training schedule from explicit model, token, state and recovery assumptions. Keep active compute time separate from availability and contingency.

Draw query-head sharing, count actual stored state and compare payloads with explicit model geometry and binary units.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

Work from training FLOPs to fleet throughput, days and GPU-hours. Then see how utilization and downtime change the estimate.

Count gradient traffic with the actual dtype, convert one-direction rates, and expose the bucket-readiness constraint that an average bandwidth division misses.

Derive GPU-hours from an explicit FLOP convention, compare Meta’s 30.84 million reported hours, and avoid mistaking the residual for a measured MFU.

Inventory a resumable checkpoint instead of rounding its bytes per parameter. Trace snapshot stall, durable drain and publication, then calculate the storage floor.

Derive ring all-reduce traffic, match directional bytes to bandwidth, and check memory feasibility before estimating the communication exposed on a training step.

Convert useful tokens into input traffic, then calculate separate checkpoint and restore budgets. Learn why an asynchronous writer can keep up with the cadence and still miss the recovery deadline.

Derive training cost from work per token, achieved model FLOPs utilization and a declared hourly price. Check the total through GPU-hours and keep project costs separate.

Estimate activation memory from saved tensor shapes and lifetimes. Use a published reference formula carefully, then check checkpointing and parallel ownership against the real peak.

Separate the all-reduce result from NCCL execution choices, and compare latency models without treating them as universal selection rules.

Choose between supported InfiniBand and RoCE designs using workload tests, congestion behavior and a complete cost model.

Follow corresponding GPU positions through a rail, including the leaf and spine links needed when one rail outgrows a switch.

Calculate queue growth during incast, then distinguish endpoint rate feedback from link-level priority pause and shared-resource contention.

Design a durable dataset and optional working-set cache around access semantics, request overhead and versioned readiness.

Measure batch readiness and producer capacity, then size prefetch for bursts without hiding a sustained throughput deficit.

Plan a reproducible tokenization build: measure CPU demand, choose a safe token dtype, publish complete shards and distinguish token lookup from training resume.

Design snapshot ownership, background writes and manifest publication separately, with explicit checkpoint bytes and recovery scope.

Construct concrete TP and DP groups, then verify physical devices and estimate traffic with per-rank rather than whole-model bytes.

Derive bandwidth from actual per-rank traffic and readiness deadlines, showing why bucketing cannot erase a capacity shortfall.

Explain packet spraying through a small path-collision model, then separate transport features from measured performance and a supported fabric purchase.

Trace a link flap through transport and job recovery. Calculate detection, restart and replay costs, and learn what continuous counters and timed tests can actually establish.

Budget exact hashes, MinHash signatures and candidate-pair growth separately. Use an executable probability model and measured filter rates to choose stage order.

Follow pause requests against the direction of data, estimate queue growth and distinguish cascade evidence from its possible causes.

Derive the two-tier endpoint limit, audit explicit 8,192-endpoint constructions and compare switches, links, optical modules and failure paths.

Investigate a firmware-correlated regression with matched metrics, mapped paths and a controlled canary. Separate a configuration hypothesis from evidence that confirms it.

Separate quota, borrowing, reclaim latency and fairness, with a demand-conserving example of when pooling improves utilization.

Keep the training state and replace the failed capacity. Trace fault detection, safe checkpoint selection, worker shutdown and restart; size spare capacity from observed failures and repair times.

First align the time window and denominator. Then attribute allocated capacity, kernel activity and useful progress to jobs before choosing between reclaim, queue policy and performance work.

Use the Llama 3 interruption record without turning job interruptions into device lifetime, predictable failures or a universal spare percentage.

Build observability around durable useful progress and aligned phase timing, showing why full-step timers can hide the late rank.

Investigate rising corrected-memory errors without promising a failure prediction, and follow recovery flags and vendor repair criteria.

Find where useful progress stopped by comparing per-rank traces, host stacks and device events. Distinguish late arrival, mismatched collectives and stalled execution before choosing a recovery action.

Derive a checkpoint interval from measured job interruptions and exposed save cost. Trace durability, lost work and recovery before treating an estimate as goodput.

Promise measurable job outcomes: time to useful work, recoverable progress and successful restoration. Define the population and exclusions before choosing targets, and give each error budget an agreed operating policy.

Read link counters with their generation-specific units and traffic exposure. Correlate changes with actual communication performance, then choose monitoring, maintenance or immediate recovery from the supported fault state.

Use one consistent timeline to separate discarded progress, recovery delay and detection. Explain the contributing conditions, then assign corrective work with owners, dates and tests that demonstrate the intended behavior.

Give every health-state transition a reason, owner and completion condition. Calibrate intervention against missed-fault risk and disruption cost, and prevent a bad signal source from removing the whole fleet.

Define which work survived, count the resources it occupied, and explain the remaining time without counting the same delay twice.

Compare the same collective across the ranks that should participate. Missing records, mismatched arguments, and unfinished operations lead to different next checks.

A successful operation can still produce a wrong value. Explain what comparisons can detect, what they miss, and how to investigate without mistaking ordinary numerical variation for corruption.

Separate removing a suspect GPU from production from proving it qualifies for replacement. Use vendor recovery requirements, device history, targeted tests, and the cost of recurrence.

Size a spare pool by separating nodes in repair from ready replacements. Work through a 2,048-node example, then check placement, recovery time and shared failure domains.

Determine what the replacement count measures, compare equivalent device exposure and investigate differences by cohort, location, age, software and policy. Job-interruption statistics cannot supply a hardware replacement baseline.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

A 128,000-ID vocabulary breaks the 16-bit storage estimate. Follow the bytes from raw ingestion to 60 TB of token payload, then design deduplication, manifests and a restartable loader.

Derive checkpoint bytes, burst and drain bandwidth; publish only complete manifests, then prove restore and retention work under failures.

A team returns to claim its GPUs, but another team is borrowing them in one indivisible job. Work through the quota tree, whole-job reclaim, recovery costs and release-to-ready budget before promising a start time.

Derive checkpoint replication and recovery budgets, identify the state a second cluster needs, and test the difference between a copied artifact and resumed training.

Detect missing and silent ranks from a roster, distinguish liveness from progress, and derive thresholds without assuming independent delays.

Allocate tasks by current dominant share and distinguish per-user fairness from cluster feasibility and physical placement.

Check useful progress before interpreting GPU activity. Run a bounded trace exercise that exposes a rank-conditional call, equal-count ordering errors and matching shapes with the wrong tensor meaning.

Explain a reliability concern with evidence, a specific proposal and an honest outcome. Show how you weighed the cost of delaying a launch and what changed afterward.

Tell an incident through the decisions you made, the evidence available at the time and the cost of recovery. Keep your role clear and distinguish mitigation from finding the root cause.

Clarify what the 256-GPU request must accomplish, then compare it with real placement and quota constraints. Offer a feasible alternative and explain who can decide when teams’ priorities conflict.

Explain the reason for an unpopular migration, choose informative rollout cohorts and describe recovery, communication and transition cost without inventing an incident.

Explain how a plausible diagnosis became too certain, what disproved it and what changed in your investigation. Separate observed symptoms from causal evidence.

Plan a useful first month by tracing a real job, reviewing incidents, helping under supervision and shipping a bounded improvement with the team.

Separate a disagreement about measurements from a real capacity shortage. Work a reclaim example, expose the missing reserve and escalate an authorized decision.

Explain what grew, which constraints appeared and how the system stayed usable. Work through metric-cardinality and device-event examples without assuming a universal failure order.

Evaluate a proposed 2,000-GPU purchase using compatible capacity, dated demand and complete alternatives. Make any gap left by reclaim visible.

Support a new responder without turning an outage into an exam. Choose an appropriate role, explain handoffs and review the decisions afterward.

Defend a specific infrastructure opinion with scoped evidence, useful arithmetic and a condition that would change your view.

Explain a decision under uncertainty through the cost of waiting, bounded exposure and evidence that changes the next action.

Turn a design disagreement into a concrete comparison. Use a gang-scheduling example to explain evidence, decision ownership and responsible follow-through.

Describe senior and staff responsibility through scope, technical judgment, influence and durable results, while recognizing that company rubrics differ.

Connect a researcher tool to the workflow it improved. Measure the whole loop, meaningful adoption and effects on other users.

Offer a first-90-days plan with useful early work and clear decision points. Let the team’s evidence and constraints determine the deliverable.

Report a capacity shortfall with assumptions, consequences, feasible options and a decision deadline. Show how one uncertain project changes the forecast.

Build a five-year infrastructure outlook from explicit assumptions, competing explanations and evidence that could change your view. Connect each prediction to a practical engineering decision.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Trace GPU credits through reservation, allocation, metering and settlement, with crash-safe ownership and a separate fairness policy.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.

Use a flexible 45-minute plan to connect requirements, a request path, capacity arithmetic, selected deep dives and a clear closing decision.

Handle an inaccessible GPU with bounded recovery and preserved evidence, without treating a reboot or repeat count as proof of root cause.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

Use DCGM fields with their actual semantics, combine device health with workload progress, and calibrate alerts on correlated fleet data.

Trace a stalled stream without unbounded telemetry, understand what sampling loses, and reconcile GPU allocation with a stated cost policy.