OpenAI AI Infrastructure Engineer interview questions

Derive the weight-only shortcut, then add KV traffic and feasible eight-GPU placement. A worked latency floor shows why batch size alone cannot establish the decode bottleneck.

A proposal for more memory bandwidth needs a workload that benefits, a budget for what it costs, and a reason to abandon it. Work through a serving-specific design without assuming all LLM work has the same bottleneck.

Follow intermediate tensors through separate and fused stages, calculate the traffic and whole-block bounds, and check spills and numerical behavior.

Trace stable row normalization, handle independent strides and fully masked inputs, then separate fusion traffic from measured speed.

Follow the running maximum and weighted sum across attention tiles, with executable arithmetic and an honest memory-traffic comparison.

Start with the operation, supported hardware and performance target. Compare ordinary Triton, lower-level Triton interfaces and CUDA using a measured gap that is large enough to justify maintaining custom code.

Follow graph capture, guards, differentiation and generated kernels. Then distinguish graph breaks from recompilation and measure whether warm execution repays compilation and startup costs.

Choose between local candidates, radix selection and sorting from k and the required output. A tested FP32 reference makes ties, signed zero, NaNs and threshold selection explicit.

Compute MFU with a stated work model, then follow the training step’s critical path across input, host dispatch, computation and communication. Rank experiments by exposed time instead of assigning universal gains to fixes.

Count gradient traffic from a named dtype, separate initialization and buffer broadcasts, and measure the reduction tail instead of assuming it is hidden.

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Separate the 100B memory floor from the training deadline, then compare two 1,024-GPU layouts with their actual state ownership and communication schedules.

Follow saved and recomputed values through backward. Compare complete update time and peak memory instead of promising a fixed memory saving or slowdown.

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.

Distinguish stale telemetry, skipped updates and a real learning plateau using counters, parameter changes and a fixed evaluation batch.

Separate a longer step from lower throughput, trace the dependency that changed, and distinguish input, workload, device and path causes. Then calculate whether a repair recovers more time than it costs.

Separate generation, scoring, learning and policy publication. Size the pools from matched workload measurements, and make policy versions explicit before overlapping rollout generation with updates.

Offline DPO trains on existing preference pairs, so online rollout and reward services leave the update loop. Account for both responses, decide whether reference scores can be cached, and retain the data-generation and evaluation costs outside training.

GRPO replaces a learned value baseline with comparisons within response groups. Size generation and verification from the actual group workload, then distinguish continuous batching from changes to the training algorithm.

Build a training schedule from explicit model, token, state and recovery assumptions. Keep active compute time separate from availability and contingency.

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Two requests share 2,000 leading tokens but ask different questions afterward. Follow the shared state through a branch, calculate the reusable blocks, and distinguish memory savings from measured latency savings.

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

Trace a dropped stream through bounded queues, engine cancellation and safe resource release; define what retries and incomplete delivery mean.

Images add preprocessing and encoder work before the language model can answer. Their effect on prompt length and cache memory depends on the model. Budget each stage, and distinguish image-embedding reuse from prefix-cache reuse.

Greedy decoding chooses the highest-scoring token, but identical prompt text does not guarantee identical inputs or numerical execution. Check the effective request and deployment, investigate the first differing token, and define the reproducibility guarantee precisely.

Translate request demand into prompt work, generated tokens and retained state, then size a measured serving configuration and its failure reserve.

Calculate Llama 3.1 70B KV cache memory at 128k context using eight KV heads. Worked BF16 and FP8 payloads, GB versus GiB, and per-GPU capacity checks.

The 6ND rule, where the 6 comes from, and the 6.3e24 that every training-time and GPU-count question in the loop is built on.

Solve for useful throughput, round to a supported parallel layout, and account for calendar time once. The right GPU count need not be a power of two.

Translate active streams into replicas using growth reservations and matched latency tests. Show peak and failure headroom separately, with no invented benchmark.

Compare ownership with the actual rental contract, using explicit cash flows, resale assumptions and the hours that can really be avoided.

Size for a tested peak and a stated failure, then price an explicit daily scaling schedule. Keep request-slot occupancy separate from GPU utilization.

Derive training cost from work per token, achieved model FLOPs utilization and a declared hourly price. Check the total through GPU-hours and keep project costs separate.

Separate quota, borrowing, reclaim latency and fairness, with a demand-conserving example of when pooling improves utilization.

Choose a sandbox from its threat model and supported workload, then calculate density from total session memory and CPU demand. GPU access needs its own isolation and compatibility review.

Build a durable, bounded scheduling loop with reservations and checkpoint-aware preemption, and avoid universal queue-wait formulas.

First align the time window and denominator. Then attribute allocated capacity, kernel activity and useful progress to jobs before choosing between reclaim, queue policy and performance work.

Build observability around durable useful progress and aligned phase timing, showing why full-step timers can hide the late rank.

Find where useful progress stopped by comparing per-rank traces, host stacks and device events. Distinguish late arrival, mismatched collectives and stalled execution before choosing a recovery action.

Promise measurable job outcomes: time to useful work, recoverable progress and successful restoration. Define the population and exclusions before choosing targets, and give each error budget an agreed operating policy.

Define request success, first-output latency and streaming continuity for each service class. Count bad outcomes against an explicit budget, then use paired long and short windows to detect significant ongoing degradation.

Use one consistent timeline to separate discarded progress, recovery delay and detection. Explain the contributing conditions, then assign corrective work with owners, dates and tests that demonstrate the intended behavior.

Stop rollout expansion, preserve evidence and use the safest available mitigation. Compare versions under representative load, and size canary exposure from the regression you need to detect rather than a fixed traffic percentage.

Follow a request from the client's connection pool to the first text on screen. Work the timing budget, locate a buffering delay and choose what to trace without losing the slow requests.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Trace GPU credits through reservation, allocation, metering and settlement, with crash-safe ownership and a separate fairness policy.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

Four result rows can still hide a missing case. Build immutable evaluation identities, retry-safe commits, versioned scoring and a capacity estimate tied to measured prompt throughput.

Divide one token budget across gateways instead of copying it. Follow output reservation and settlement, then handle grants, retries and outages without inventing extra burst capacity.

Remove the largest region from a concrete capacity plan. Check the surviving budget, tenant eligibility, cold prefixes and retry semantics before claiming the service can fail over.

Separate durable conversation state from live generation, bound streaming buffers, and make reconnect, cancellation and free-tier budgets explicit.

Trace a stalled stream without unbounded telemetry, understand what sampling loses, and reconcile GPU allocation with a stated cost policy.

Separate session state, model calls and tool execution. Calculate growing context, budget isolated workspaces and recover uncertain actions without duplicating side effects.

Follow a video job from prompt to downloadable artifact. Work the latent-shape arithmetic, compare batching with actual delivery times, and design bounded queues, cancellation and safe retries.

A team returns to claim its GPUs, but another team is borrowing them in one indivisible job. Work through the quota tree, whole-job reclaim, recovery costs and release-to-ready budget before promising a start time.

Trace usage from execution to one settled charge. Distinguish requests, attempts and events, then handle cancellation, late records, replay and reconciliation.

Size a hypothetical trillion-parameter MoE from its weight placement, attention-state layout and expert traffic. Use rack specifications as bounds, then measure the serving workload before promising throughput.

Separate total and active parameters, then size persistent state, approximate compute and token dispatch. Compare supported parallel layouts using per-rank memory, measured communication and recovery.

Build a small decimal billing calculator with an explicit rounding policy, exact durations and tests for invalid records and credits.

Two parameters, one lazy refill and one lock. What capacity and rate each control and why conflating them is the usual bug, the clock choice that avoids a whole class of failure, and the retry-after value that turns a rejection into something a client can act on.

The union is a sort and a scan. The intersection is the same sweep with a counter, and it is the one that goes wrong, because a single GPU reporting overlapping intervals will be counted twice and produce an answer that looks plausible. Both implementations, the trap, and the tests that catch it.

Implement bounded retries with full jitter, a shared credit budget and a cooperative deadline. Test repeatability and explain what a timeout leaves uncertain.

Per key, an ascending list of versions and a parallel list of values, with reads doing a binary search. The three cases that decide whether the design is right: a key that did not exist yet, a key that was deleted, and a version at which some other key was written.

Implement priority admission and completion, preserve a preempted job’s sequence, and distinguish a sufficient greedy victim set from an optimal one.

Implement a bounded producer-consumer pipeline with explicit shutdown and error reporting. Keep work outside collection locks and account for memory beyond the input queue.

Decode fragmented UTF-8 with an incremental codec. Test malformed bytes, truncated endings and the boundary between byte decoding and event parsing.

Implement stable temperature, top-k and top-p sampling with explicit tie rules, masks, invalid inputs and probability tests.

Implement bounded recovery probes, state-generation tickets and a recent failure window. Test concurrency and late completions.

Design job creation, cancellation and listing around retries, durable ownership and an explicit pagination contract.

Allocate tasks by current dominant share and distinguish per-user fairness from cluster feasibility and physical placement.

Check useful progress before interpreting GPU activity. Run a bounded trace exercise that exposes a rank-conditional call, equal-count ordering errors and matching shapes with the wrong tensor meaning.

Explain a reliability concern with evidence, a specific proposal and an honest outcome. Show how you weighed the cost of delaying a launch and what changed afterward.

Connect an infrastructure problem you want to solve with work you have actually done. Explain what prompted the move now, what transfers from your background and what you still need to learn.

Clarify what the 256-GPU request must accomplish, then compare it with real placement and quota constraints. Offer a feasible alternative and explain who can decide when teams’ priorities conflict.

Connect your view on AI safety to systems you could build: model access, audit evidence, evaluation capacity and a tested way to stop or reverse a release. Explain the costs and limits of each control.

Separate a disagreement about measurements from a real capacity shortage. Work a reclaim example, expose the missing reserve and escalate an authorized decision.

Handle a request to skip a safety check by identifying the required gate, finding approved alternatives and making exception authority explicit.

Explain what grew, which constraints appeared and how the system stayed usable. Work through metric-cardinality and device-event examples without assuming a universal failure order.

Defend a specific infrastructure opinion with scoped evidence, useful arithmetic and a condition that would change your view.

Explain a decision under uncertainty through the cost of waiting, bounded exposure and evidence that changes the next action.

Offer a first-90-days plan with useful early work and clear decision points. Let the team’s evidence and constraints determine the deliverable.

Report a capacity shortfall with assumptions, consequences, feasible options and a decision deadline. Show how one uncertain project changes the forecast.

Build a five-year infrastructure outlook from explicit assumptions, competing explanations and evidence that could change your view. Connect each prediction to a practical engineering decision.

Derive resident-block limits and distinguish resident warps from ready work before trading registers for more occupancy.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Draw query-head sharing, count actual stored state and compare payloads with explicit model geometry and binary units.

Follow a four-GPU request from advertised capacity to concrete devices, and see why free GPUs across nodes cannot satisfy one pod.

DRA gives the scheduler device claims and attributes to work with. Compare that with extended-resource counts, then check the deployed driver and controller support before promising sharing, partitioning or gang placement.

Use a flexible 45-minute plan to connect requirements, a request path, capacity arithmetic, selected deep dives and a clear closing decision.