Hard AI infra interview questions

Separate Blackwell chip gains from the 72-GPU domain, follow a token to its expert owners, and test placement instead of dividing model bytes by 72.

Carry compute and memory reasoning to TPU v6e, correct all-gather ownership arithmetic, and learn the compilation, sharding and custom-kernel workflow.

Trace a two-by-two output-stationary array cycle by cycle. Derive its fill and drain cost without turning a toy schedule into a TPU benchmark.

Compare actual useful work and memory units, validate Neuron support, and calculate when rental savings repay porting and operations.

See how 192 GB changes replica shape, calculate KV headroom with exact units, and compare feasible deployments at equal quality and latency.

Follow a dependent read path to understand SRAM locality, aggregate bandwidth and the difference between one fast stream and total serving throughput.

Quantize a sixteen-value block, count the scale bytes, and use paired quality tests to decide which tensors need higher precision.

Derive a 16-leaf, 8-spine fabric from 512 endpoints, trace local and cross-leaf routes, and check what rail placement and oversubscription actually guarantee.

Use a profiler summary to form a memory-traffic hypothesis, then check counters, timing and useful work before choosing an optimization.

Calculate the data a kernel could reuse, then test whether it stays in L2. Distinguish capacity, cache-policy hints and measured traffic from a guarantee of residency.

Follow a host buffer across a NUMA boundary, calculate the cost with declared bandwidths, and align CPU workers and memory with the GPU that consumes the data.

Compare the cost of serving the same workload at the same latency and quality. A memory-traffic estimate explains what each GPU might improve; measured throughput and your actual rental price determine the purchase.

Separate valid 2:4 structure, packed representation and sparse dispatch from model quality and the fraction of request time that can improve.

Derive cooling flow for a 4.8 MW hall, separate liquid and air loads, and test CDU capacity and redundancy at the actual design conditions.

Localize a slow fabric rail with matched tests, path comparisons and counter deltas. Separate an independent-bandwidth sum from collective completion time.

Define acceptance with component and system tests, absolute targets, peer comparisons and documented retests. Interpret burn-in exposure without treating it as a reliability guarantee.

Investigate rack power events at the correct electrical boundary and time scale. Test load reductions without assuming GPU caps control every watt or transient.

Diagnose a multi-node regression by comparing host and GPU transfer paths, active registration mechanisms and the tested software stack.

Design a management path that survives specified production failures. Count all endpoint classes, restrict device authority and test the complete recovery route.

Build a deployment schedule from dependencies and committed lead times. Calculate float and price the delay only when an action moves usable capacity earlier.

Investigate widespread acceptance failures using matched controls, valid thresholds and cohort denominators. Preserve hardware and shared causes as testable hypotheses.

Count endpoints, allocate every switch port once, and compare a common fabric with independent rails before adding a third tier.

Compare 64 or 72 useful GPUs, identify the communication that could improve, and test whether a larger NVLink domain justifies the operational requirements.

Budget the full megawatt, fit whole servers, and compare useful training throughput under measured power limits. A worked cap example shows when more GPUs help.

Choose a supported tenant boundary, secure the surrounding services, and verify cleanup before reassignment. Calculate fragmentation from actual requests.

Reproduce the baseline, compare equivalent work, and trace the missing time. Acceptance, production and the original plan may differ in more than one way.

Protect running work, validate a supported upgrade bundle, and expand through measured cohorts with a tested recovery path.

Keep the full failure history and investigate what changed. One passing retry is evidence, but it is not a repair or an acceptance policy.

Survey real facility headroom, phase work around shared dependencies, and compare retrofit with relocation using useful capacity and total transition cost.

Confirm the event definition and exposure, compare cohort rates, then test whether the cause follows a module, location or optical path.

Compare complete fabric proposals on workload performance, support and total cost. Translate any measured regression into application impact before pricing the discount.

Follow an accumulator across K tiles, count reuse at each memory level, and decide whether a tile improves measured GEMM performance.

Compare the mechanisms in FlashAttention versions 1, 2 and 3: IO savings, work partitioning, asynchronous dependencies and FP8 accuracy.

Separate cache activity from DRAM traffic, check the access width and active lanes, and use a controlled layout change to test a profiler hypothesis. A few percentages cannot predict the fixed kernel’s runtime.

Use a timeline to establish launch overhead, then trace what graph replay changes. Stable buffers, page-table updates, capture-safe operations and batch buckets determine whether the serving engine can use it safely.

Follow a logical token through the block table into a physical KV page. Then combine split attention states correctly, including empty splits, and count lookup traffic at the layer and head the kernel actually processes.

Use cuBLAS or cuBLASLt when its supported operation fits. CUTLASS exposes kernel building blocks for custom layouts, fusion and scheduling, but each choice needs a whole-operation benchmark and an ownership plan.

Build a numerical reference, count the traffic and investigate a row-wise fused kernel. Separate bandwidth lower bounds from measured performance.

Verify a drafted prefix without biasing the target distribution, then roll back rejected KV state. A six-token prefix exposes the off-by-one: nine committed tokens can have only eight valid KV entries.

Derive when operand scales can be applied after a dot product and when they must follow its K blocks. Separate integer zero points, floating-point range, scale granularity and accumulator behavior.

Separate per-expert submission, small-grid scheduling and partially filled tiles. A grouped kernel can share work across experts, but routing, packing and output combination still determine the layer’s cost.

Compute MFU with a stated work model, then follow the training step’s critical path across input, host dispatch, computation and communication. Rank experiments by exposed time instead of assigning universal gains to fixes.

HIP can translate many CUDA interfaces, but it cannot validate the assumptions inside a kernel. Check thread-group widths, masks, shared state and numerical behavior before tuning the MI300X implementation.

Define what the timer includes, give both implementations the same conditions, and keep the sample distribution. A useful benchmark explains the workload and comparison well enough for someone else to reproduce them.

Derive row-major offsets, broadcast the bias correctly, and compare a lane-by-lane CPU reference with a vector expression. Explain when shared memory adds no reuse.

A canceled request returns its buffers to a pool while device work is still outstanding. Trace ownership through two streams and design a safe reclamation path.

Separate compilation, module loading and execution before judging a faster kernel. Define the artifact and startup evidence needed to release a compiler upgrade.

Derive the ideal gain from filling a partial wave, then account for partial outputs, reduction order, workspace and the epilogue before selecting split-K.

Trace local counts into globally disjoint output ranges. Find the missing block carry and explain why a spin-wait barrier can deadlock the grid.

Review the input contract, adversarial cases, evaluator independence and compiled baseline before trusting a generated GPU kernel or its reported speedup.

Trace a causal query across local and remote KV blocks. Separate sequence and context sharding, count one concrete tensor, and avoid dividing memory twice.

Follow one token to two expert ranks and back, then count logical assignments separately from physical network transfers.

Use a ten-expert example to separate assignments, rank load and dropped routes, then compare balancing controls against quality.

Construct a 16384-GPU candidate layout from ownership and batch constraints, with explicit sharding recipes and pipeline idle time.

Distinguish stale telemetry, skipped updates and a real learning plateau using counters, parameter changes and a fixed evaluation batch.

Balance checkpoint pauses against lost work using job-level interruption data, while accounting for asynchronous durability lag.

Treat recovery as a new worker generation, then compare spare replacement with a legal shrink and its batch consequences.

Separate a longer step from lower throughput, trace the dependency that changed, and distinguish input, workload, device and path causes. Then calculate whether a repair recovers more time than it costs.

Overlap starts with a dependency: communication must finish before its consumer needs the result. Estimate the available window, then measure readiness, contention and complete step time for DDP, FSDP or tensor parallelism.

A global-array program lets the compiler derive communication from sharding choices. Work through a small matrix example, then separate that programming model from TPU topology, compilation and Pathways recovery.

Separate packed base weights, quantization metadata and trainable adapter state. Count the real projection shapes, then leave room for activations and temporary work before calling a one-card configuration feasible.

Separate generation, scoring, learning and policy publication. Size the pools from matched workload measurements, and make policy versions explicit before overlapping rollout generation with updates.

GRPO replaces a learned value baseline with comparisons within response groups. Size generation and verification from the actual group workload, then distinguish continuous batching from changes to the training algorithm.

Separate FP8 input quantization from accumulation error. A small representable-value example shows when a shared scale loses small values, while the DeepSeek-V3 report supplies one carefully scoped training recipe.

Describe logical tensors independently of writer ranks, publish only a complete checkpoint, and reconstruct the target layout from stored chunks. Asynchronous saving reduces blocking but still needs a consistent snapshot and a memory budget.

Version teacher outputs, separate learning from evaluation, and catch a release whose domain score improves while a protected capability regresses.

Three routes with different costs: keep PyTorch through XLA, bet on the native backend, or adopt the JAX reference trainer. Choose from the model's custom code and the timeline, then prove parity before scaling.

Compare a complete phase handoff with a tuned combined pool, then test the queueing reversal and qualified completions. Nominal link speed and fleet size alone cannot decide the split.

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

Filter engine candidates by exact compatibility, then compare correctness, goodput and operational cost on the same workload.

Follow adapter-specific token rows through a shared base, calculate stored adapter bytes, and separate the resident set from the active batch.

Forecast demand across startup delay, track replicas already being created, and drain slowly enough to avoid repeating the same scale decision.

Trace a model startup through storage, runtime setup and device loading. Transfer floors and dependencies determine whether a minute is plausible.

Separate checkpoint memory, expert assignments and retained MLA state. The active parameter count alone cannot predict per-rank fit or token latency.

A cached replica can still be the slower choice. Compare saved prefill against waiting time, then handle stale cache reports, pending dispatches and tenant boundaries.

Calculate retained cache and prefill work for a stated 128k model shape, then choose scheduling and memory policies from measured limits.

Compare proposal methods by committed output over complete round time, with explicit sampling guarantees and target compatibility.

Build a measured first-token budget and test the smallest serving design that meets it, including cold prompts, queueing and delivery.

Images add preprocessing and encoder work before the language model can answer. Their effect on prompt length and cache memory depends on the model. Budget each stage, and distinguish image-embedding reuse from prefix-cache reuse.

A conversation's cache is leaving GPU memory. Work out whether to save it, reload it later, or recompute, including transfer queues and the chance that nobody returns.

Price duplicate work before racing replicas. Separate independent stragglers from shared overload, and define what happens when a winner starts streaming.

Choose the release boundary, account for buffering delay, and keep policy versions attached to decisions. Extend the design to queued review and appeals.

Trace document revisions, cancellation and deadlines so a late correct answer cannot overwrite newer text. Size useful suggestions rather than every keystroke.

Diagnose a stream that begins quickly but produces useful answer content late. Compare generation, parser and client clocks, then test a reasoning budget without hiding truncated or incorrect responses.

Find whether expert placement, small batches or communication makes the deployment slower. Compare equivalent workloads and inspect the worker that holds up each step.

Use the actual tensor index to size Kimi K3, then separate a published hardware starting point from production capacity. Account for mixed precision, hybrid recurrent state, parser behavior, and the workload behind a throughput claim.

Check the bound’s assumptions, locate the critical path, and quantify which optimization can actually improve one user’s decode latency.

Long prefills coincide with decode stalls. Compare a tuned combined pool with separate phases, derive transfer and pool capacity, and test whether the improvement survives connector failures.

The nodes joined the cluster, but the first request hangs. Trace how far it got, check worker assignments, and reproduce the blocked operation on the same communication path.

Compare committed output per complete draft-and-verify cycle with ordinary decoding. Derive the acceptance-length model, separate it from engine counters, and test latency, goodput and memory across the actual workload.

A fourfold ratio is a symptom, not a diagnosis. Trace individual requests, separate waiting from execution and delivery, and test workload, scheduling, cache and capacity hypotheses against the service objective.

Build a week around release gates and dependencies. Budget artifact transfer, validate the exact runtime path, measure the intended workload, and size the canary evidence before promising full rollout.

A context-window limit is not a latency or concurrency guarantee. Budget retained state, prefill and shared service time; distinguish cache reuse from eliminating work; choose isolation only when measured interference justifies its cost.

Design a shared catalogue with explicit service tiers, reproducible deployments and bounded cold starts. Residency depends on burst behavior and resource reuse; placement depends on GPU groups and interference as well as memory.

Verify the throughput definition and service outcomes, compare resolved configurations and execution traces, then isolate supported changes. Quantify capacity replacement correctly and choose rollback from the full release risk.

Budget growing attention state, bounded recurrent state and retained prefix snapshots separately. Work the short-context crossover, then test whether the cache manager resumes every layer from the same prefix boundary.

Define whether half the footprint means checkpoint bytes or runtime memory. Choose a supported recipe, measure which tensors it changes, separate calibration from held-out evaluation, and validate quality and serving cost.

Test the complete serving bundle with contract fixtures, representative tasks and load-aware outcomes. Preserve paired results, calibrate the scorer and distinguish a relative improvement from meeting an absolute product requirement.

Attribute queueing, token work and live state before choosing a cap. Bound new admission, verify recovery, and distinguish fair scheduling, reserved service and physical isolation without promising instant or absolute protection.

Measure exposed launch and dispatch delay, actual graph coverage, capture memory and warmup. Batch size influences the result, but does not alone determine whether graphs improve the service.

Group compatible adapters by pinned base, derive their size from target matrices and ranks, and benchmark mixed batches. Catalogue size, CPU residency, GPU slots and active adapters are separate limits; base sharing does not make fifty variants operationally free.

Separate tensor format, model semantics and runtime support. Budget weights plus context state, and check conversion before judging a quantized artifact.

Trace graph rewrites, provider placement and shape profiles. Use a fallback counterexample to show why fewer operators can still produce higher latency.

Turn a benchmark number into a bill that survives real utilization, then write the serving contract the gateway enforces and the evaluation the tuned configuration still owes.

Same model, same engine name, different execution model. Trace what moves into compilation, how shape buckets trade compile time for padding, and how per-device memory decides the slice.

Translate active streams into replicas using growth reservations and matched latency tests. Show peak and failure headroom separately, with no invented benchmark.

Separate resident parameters, active work and the union of experts touched by a batch. Compare toy dense and MoE budgets without promising a fixed speedup.

Compare ownership with the actual rental contract, using explicit cash flows, resale assumptions and the hours that can really be avoided.

Convert useful tokens into input traffic, then calculate separate checkpoint and restore budgets. Learn why an asynchronous writer can keep up with the cadence and still miss the recovery deadline.

Size for a tested peak and a stated failure, then price an explicit daily scaling schedule. Keep request-slot occupancy separate from GPU utilization.

Derive training cost from work per token, achieved model FLOPs utilization and a declared hourly price. Check the total through GPU-hours and keep project costs separate.

Estimate a TP8 decode step from weights, cache reads and explicitly assumed collective times, then replace the serial worksheet with a measured critical path.

Derive the batch-295 shortcut, count the tensors it omits, and test whether the resulting kernel estimate is even inside the replica’s memory budget.

Estimate activation memory from saved tensor shapes and lifetimes. Use a published reference formula carefully, then check checkpointing and parallel ownership against the real peak.

Use Little’s law to distinguish waiting and active requests, then test saturation with a stated service model. Avoid turning a mean occupancy identity into a universal latency rule.

Use bandwidth per dollar as a first comparison, then check feasible memory, supported execution and complete service cost before ranking accelerators.

Count endpoint, leaf, aggregation and core ports for an explicit 4,096-GPU design. Compare 1:1 and 2:1 without confusing switch ports, links or optics.

Calculate queue growth during incast, then distinguish endpoint rate feedback from link-level priority pause and shared-resource contention.

Plan a reproducible tokenization build: measure CPU demand, choose a safe token dtype, publish complete shards and distinguish token lookup from training resume.

Design snapshot ownership, background writes and manifest publication separately, with explicit checkpoint bytes and recovery scope.

Construct concrete TP and DP groups, then verify physical devices and estimate traffic with per-rank rather than whole-model bytes.

Explain packet spraying through a small path-collision model, then separate transport features from measured performance and a supported fabric purchase.

Derive a complete rail topology and distinguish physical links, switch ports and transceivers. Size storage and management from their own requirements.

Trace a link flap through transport and job recovery. Calculate detection, restart and replay costs, and learn what continuous counters and timed tests can actually establish.

Budget exact hashes, MinHash signatures and candidate-pair growth separately. Use an executable probability model and measured filter rates to choose stage order.

Follow expert assignments from logical payload to off-rank and cross-node traffic. Compare dispatch and combine dtypes, receiver skew and local-domain placement.

Follow pause requests against the direction of data, estimate queue growth and distinguish cascade evidence from its possible causes.

Derive the two-tier endpoint limit, audit explicit 8,192-endpoint constructions and compare switches, links, optical modules and failure paths.

Two tenants can fit the average storage budget and still miss both checkpoint deadlines. Work through reservations, borrowing and bounded admission while protecting training reads, then test the design under failure.

Define warm, cached and truly cold paths, then size transfer budgets and warm capacity without promising universal sub-second restores.

Choose a sandbox from its threat model and supported workload, then calculate density from total session memory and CPU demand. GPU access needs its own isolation and compatibility review.

Size usable inference headroom from the complete readiness delay and demand ramp, then decide which training capacity can safely be borrowed.

Build a durable, bounded scheduling loop with reservations and checkpoint-aware preemption, and avoid universal queue-wait formulas.

Count feasible placements for the waiting job, including its per-node and topology requirements. Then compare waiting, reservations and checkpoint/restart consolidation using their actual disruption costs.

Keep the training state and replace the failed capacity. Trace fault detection, safe checkpoint selection, worker shutdown and restart; size spare capacity from observed failures and repair times.

Choose the tenant boundary before planning base-model reuse. Account for frozen weights, adapter state and activations separately, then schedule by compatibility, fairness and the real cost of restarting training.

First align the time window and denominator. Then attribute allocated capacity, kernel activity and useful progress to jobs before choosing between reclaim, queue policy and performance work.

Calculate traffic across the actual inter-cloud boundary before deciding. Whole-job placement often avoids repeated transfers, but communication volume, overlap, convergence requirements and available links determine whether a split job is viable.

A fabric address, an RDMA device and a working GPU-memory registration path are separate requirements. Verify each inside the pod, then test the actual GPU-to-NIC placement and collective behavior.

Treat the upgrade as a tested change to a compatibility set. Drain safely, canary representative workloads, size waves from usable spare capacity and rehearse restoration of the previous working configuration.

Find where useful progress stopped by comparing per-rank traces, host stacks and device events. Distinguish late arrival, mismatched collectives and stalled execution before choosing a recovery action.

Read link counters with their generation-specific units and traffic exposure. Correlate changes with actual communication performance, then choose monitoring, maintenance or immediate recovery from the supported fault state.

Use one consistent timeline to separate discarded progress, recovery delay and detection. Explain the contributing conditions, then assign corrective work with owners, dates and tests that demonstrate the intended behavior.

Give every health-state transition a reason, owner and completion condition. Calibrate intervention against missed-fault risk and disruption cost, and prevent a bad signal source from removing the whole fleet.

Compare the same collective across the ranks that should participate. Missing records, mismatched arguments, and unfinished operations lead to different next checks.

Measure throughput at several power limits, translate the facility target into a GPU budget, and account for slower ranks before changing the fleet.

Loss of cooling can outpace human response. Use a simple heat calculation to explain why protection must be automatic, then distinguish that illustration from the installed system’s validated behavior.

Separate removing a suspect GPU from production from proving it qualifies for replacement. Use vendor recovery requirements, device history, targeted tests, and the cost of recurrence.

Size a spare pool by separating nodes in repair from ready replacements. Work through a 2,048-node example, then check placement, recovery time and shared failure domains.

Follow cache hits and misses into shard calls, rerank pairs and generation work; size consistent branches and enforce access-aware reuse.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

Eight ready slots meet a twelve-request burst. See why starting instances do not count as ready capacity, then design caches, isolation, admission and a measurable startup contract.

A 128,000-ID vocabulary breaks the 16-bit storage estimate. Follow the bytes from raw ingestion to 60 TB of token payload, then design deduplication, manifests and a restartable loader.

A reusable prefix was evicted from GPU memory. Compare host or NVMe reload with recomputation, including queues, then design safe promotion, eviction and recovery across the tiers.

Remove the largest region from a concrete capacity plan. Check the surviving budget, tenant eligibility, cold prefixes and retry semantics before claiming the service can fail over.

Derive checkpoint bytes, burst and drain bandwidth; publish only complete manifests, then prove restore and retention work under failures.

Follow GPU ownership from reservation through verified delivery and cleanup, with stale-operation fencing, recovery and auditable metering.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.

Separate session state, model calls and tool execution. Calculate growing context, budget isolated workspaces and recover uncertain actions without duplicating side effects.

Follow a video job from prompt to downloadable artifact. Work the latent-shape arithmetic, compare batching with actual delivery times, and design bounded queues, cancellation and safe retries.

Compare the output of a speculative round with drafting and verification cost. Work the break-even condition, then design a measured rollout with memory and correctness checks.

Derive checkpoint replication and recovery budgets, identify the state a second cluster needs, and test the difference between a copied artifact and resumed training.

Size both serving pools from workload-matched rates, then check memory, shared transfer paths and surviving capacity. Follow admission and buffer ownership through failure and scaling.

Separate total and active parameters, then size persistent state, approximate compute and token dispatch. Compare supported parallel layouts using per-rank memory, measured communication and recovery.

Connect POST-and-poll semantics to durable item results, retry ownership and deadline capacity. Explain how cheap batch execution can still miss the delivery contract.

Find concrete failure paths in a small serving architecture. Rank correctness, admission and placement fixes before proposing additional infrastructure.

Trace event time and serving availability through a late-arriving feature. Build an executable historical join and keep online freshness separate from training correctness.

Build a reference-counted KV block allocator and test that forked sequences keep their data independent when either branch appends.

Simulate admission, decode growth and preemption. Check memory accounting and useful progress with the same finite workload.

Implement a stable hash ring, test which keys move, and distinguish balanced ownership from balanced serving traffic.

Implement owner-pop and thief-steal semantics with correct accounting for running tasks that generate children.

Allocate tasks by current dominant share and distinguish per-user fairness from cluster feasibility and physical placement.

Build a complete segmented metrics-store reference with duplicate-safe range queries, visible retention and tested compaction.

Apply a learned merge order without retraining on the input. Test overlapping pairs, preserve UTF-8 bytes, and turn tokenizer differences into a capacity estimate.

Keep live and completed candidates separate, score whole sequences, and trace a case where greedy decoding loses. Explain the cache cost of branching.

Follow the matrix shapes through a low-rank update. Verify merged inference numerically and show why multiplying sums of adapter factors gives the wrong weights.

Build a reverse-mode graph for addition and multiplication. Use a repeated input to expose overwritten gradients, then check the derivative numerically.

Keep request timing independent, make snapshots consistent, and expose overwritten records. Test the ring after more concurrent writes than it can retain.

Define when expiry is observed, stage writes privately, and reject a conflicting commit. Use a small conservative protocol before optimizing concurrency.

Build a small CPU reference, overwrite a bounded KV ring, and compare every step with a recomputed attention window. Keep model semantics separate from eviction.

Derive the scale, handle zero tensors, clamp before casting, and separate rounding error from clipping. Compare one outlier under tensor and channel scales.

Handle a request to skip a safety check by identifying the required gate, finding approved alternatives and making exception authority explicit.

Explain what grew, which constraints appeared and how the system stayed usable. Work through metric-cardinality and device-event examples without assuming a universal failure order.

Evaluate a proposed 2,000-GPU purchase using compatible capacity, dated demand and complete alternatives. Make any gap left by reclaim visible.

Defend a specific infrastructure opinion with scoped evidence, useful arithmetic and a condition that would change your view.

Describe senior and staff responsibility through scope, technical judgment, influence and durable results, while recognizing that company rubrics differ.

Offer a first-90-days plan with useful early work and clear decision points. Let the team’s evidence and constraints determine the deliverable.

Explain an isolation finding through the intended boundary, authorized evidence, designated reporting process and checks for both denied and permitted access.

Report a capacity shortfall with assumptions, consequences, feasible options and a decision deadline. Show how one uncertain project changes the forecast.