Anthropic AI Infrastructure Engineer interview questions

Count two matrix shapes to derive the H100 ridge, throughput ceilings and time floors. Compare an illustrative measurement with its bound before deciding what to optimize.

Derive the weight-only shortcut, then add KV traffic and feasible eight-GPU placement. A worked latency floor shows why batch size alone cannot establish the decode bottleneck.

A proposal for more memory bandwidth needs a workload that benefits, a budget for what it costs, and a reason to abandon it. Work through a serving-specific design without assuming all LLM work has the same bottleneck.

Follow the running maximum and weighted sum across attention tiles, with executable arithmetic and an honest memory-traffic comparison.

Follow graph capture, guards, differentiation and generated kernels. Then distinguish graph breaks from recompilation and measure whether warm execution repays compilation and startup costs.

Build a numerical reference, count the traffic and investigate a row-wise fused kernel. Separate bandwidth lower bounds from measured performance.

Compute MFU with a stated work model, then follow the training step’s critical path across input, host dispatch, computation and communication. Rank experiments by exposed time instead of assigning universal gains to fixes.

Define what the timer includes, give both implementations the same conditions, and keep the sample distribution. A useful benchmark explains the workload and comparison well enough for someone else to reproduce them.

Longer sequences exercise more state updates, different boundaries and sometimes different kernel paths. Use a two-tile example to check online softmax, then test masking, indexing and precision against a stable reference.

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Compute MFU from a declared global token count and time interval. Check the accounting before judging the percentage or choosing an optimization.

Separate the 100B memory floor from the training deadline, then compare two 1,024-GPU layouts with their actual state ownership and communication schedules.

Count tensor-parallel messages and compare their payload service time on two explicit paths. Explain why TP8 is a common placement choice, not a fixed ceiling.

Trace a causal query across local and remote KV blocks. Separate sequence and context sharding, count one concrete tensor, and avoid dividing memory twice.

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.

Distinguish stale telemetry, skipped updates and a real learning plateau using counters, parameter changes and a fixed evaluation batch.

Balance checkpoint pauses against lost work using job-level interruption data, while accounting for asynchronous durability lag.

Separate a longer step from lower throughput, trace the dependency that changed, and distinguish input, workload, device and path causes. Then calculate whether a repair recovers more time than it costs.

Separate generation, scoring, learning and policy publication. Size the pools from matched workload measurements, and make policy versions explicit before overlapping rollout generation with updates.

Offline DPO trains on existing preference pairs, so online rollout and reward services leave the update loop. Account for both responses, decide whether reference scores can be cached, and retain the data-generation and evaluation costs outside training.

Turn the daily loss into recovery and replay costs, then identify which part is measured and which is assumed. Compare checkpoint cadence, recovery work and failure prevention without promising one universal floor.

Build a training schedule from explicit model, token, state and recovery assumptions. Keep active compute time separate from availability and contingency.

Three routes with different costs: keep PyTorch through XLA, bet on the native backend, or adopt the JAX reference trainer. Choose from the model's custom code and the timeline, then prove parity before scaling.

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Trace a 20,000-token prompt through a shared token budget, including the last partial chunk. Then choose a setting from measured first-token waits and streaming gaps.

Trace an accepted prefix, a rejection and the correction token; derive emitted progress and compare it with a measured round cost.

Compare a complete phase handoff with a tuned combined pool, then test the queueing reversal and qualified completions. Nominal link speed and fleet size alone cannot decide the split.

Two requests share 2,000 leading tokens but ask different questions afterward. Follow the shared state through a branch, calculate the reusable blocks, and distinguish memory savings from measured latency savings.

Preserve document identity through packed prompts and completion reordering. Derive rounded cache reservations, then measure feasible wave sizes against the API contract.

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

A cached replica can still be the slower choice. Compare saved prefill against waiting time, then handle stale cache reports, pending dispatches and tenant boundaries.

Calculate retained cache and prefill work for a stated 128k model shape, then choose scheduling and memory policies from measured limits.

Greedy decoding chooses the highest-scoring token, but identical prompt text does not guarantee identical inputs or numerical execution. Check the effective request and deployment, investigate the first differing token, and define the reproducibility guarantee precisely.

Translate request demand into prompt work, generated tokens and retained state, then size a measured serving configuration and its failure reserve.

Define cold and reused-context objectives, verify the checkpoint's real state layout, and size per-rank memory and transfer demand. Caching and phase separation help only when their compatibility, retention and capacity assumptions hold.

Test the complete serving bundle with contract fixtures, representative tasks and load-aware outcomes. Preserve paired results, calibrate the scorer and distinguish a relative improvement from meeting an absolute product requirement.

Same model, same engine name, different execution model. Trace what moves into compilation, how shape buckets trade compile time for padding, and how per-device memory decides the slice.

Calculate Llama 3.1 70B KV cache memory at 128k context using eight KV heads. Worked BF16 and FP8 payloads, GB versus GiB, and per-GPU capacity checks.

Work from training FLOPs to fleet throughput, days and GPU-hours. Then see how utilization and downtime change the estimate.

Solve for useful throughput, round to a supported parallel layout, and account for calendar time once. The right GPU count need not be a power of two.

Work through projection and attention FLOPs for an 8,192-token prompt. Separate a peak-based lower bound, an assumed-rate estimate and observed TTFT.

Size for a tested peak and a stated failure, then price an explicit daily scaling schedule. Keep request-slot occupancy separate from GPU utilization.

Derive training cost from work per token, achieved model FLOPs utilization and a declared hourly price. Check the total through GPU-hours and keep project costs separate.

Estimate a TP8 decode step from weights, cache reads and explicitly assumed collective times, then replace the serial worksheet with a measured critical path.

Measure batch readiness and producer capacity, then size prefetch for bursts without hiding a sustained throughput deficit.

Plan a reproducible tokenization build: measure CPU demand, choose a safe token dtype, publish complete shards and distinguish token lookup from training resume.

Budget exact hashes, MinHash signatures and candidate-pair growth separately. Use an executable probability model and measured filter rates to choose stage order.

Separate quota, borrowing, reclaim latency and fairness, with a demand-conserving example of when pooling improves utilization.

Choose a sandbox from its threat model and supported workload, then calculate density from total session memory and CPU demand. GPU access needs its own isolation and compatibility review.

Size usable inference headroom from the complete readiness delay and demand ramp, then decide which training capacity can safely be borrowed.

First align the time window and denominator. Then attribute allocated capacity, kernel activity and useful progress to jobs before choosing between reclaim, queue policy and performance work.

Build observability around durable useful progress and aligned phase timing, showing why full-step timers can hide the late rank.

Find where useful progress stopped by comparing per-rank traces, host stacks and device events. Distinguish late arrival, mismatched collectives and stalled execution before choosing a recovery action.

Define request success, first-output latency and streaming continuity for each service class. Count bad outcomes against an explicit budget, then use paired long and short windows to detect significant ongoing degradation.

Use one consistent timeline to separate discarded progress, recovery delay and detection. Explain the contributing conditions, then assign corrective work with owners, dates and tests that demonstrate the intended behavior.

Define which work survived, count the resources it occupied, and explain the remaining time without counting the same delay twice.

Build an incident-review process that leads to verified changes. Set review thresholds, budget the work and interpret recurrence without mistaking a metric for proof.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.

Follow cache hits and misses into shard calls, rerank pairs and generation work; size consistent branches and enforce access-aware reuse.

Four result rows can still hide a missing case. Build immutable evaluation identities, retry-safe commits, versioned scoring and a capacity estimate tied to measured prompt throughput.

A 128,000-ID vocabulary breaks the 16-bit storage estimate. Follow the bytes from raw ingestion to 60 TB of token payload, then design deduplication, manifests and a restartable loader.

Divide one token budget across gateways instead of copying it. Follow output reservation and settlement, then handle grants, retries and outages without inventing extra burst capacity.

Remove the largest region from a concrete capacity plan. Check the surviving budget, tenant eligibility, cold prefixes and retry semantics before claiming the service can fail over.

Derive checkpoint bytes, burst and drain bandwidth; publish only complete manifests, then prove restore and retention work under failures.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.

Separate session state, model calls and tool execution. Calculate growing context, budget isolated workspaces and recover uncertain actions without duplicating side effects.

A team returns to claim its GPUs, but another team is borrowing them in one indivisible job. Work through the quota tree, whole-job reclaim, recovery costs and release-to-ready budget before promising a start time.

Derive checkpoint replication and recovery budgets, identify the state a second cluster needs, and test the difference between a copied artifact and resumed training.

Separate total and active parameters, then size persistent state, approximate compute and token dispatch. Compare supported parallel layouts using per-rank memory, measured communication and recovery.

Two parameters, one lazy refill and one lock. What capacity and rate each control and why conflating them is the usual bug, the clock choice that avoids a whole class of failure, and the retry-after value that turns a rejection into something a client can act on.

Implement size- and timer-triggered batching, then test cancellation, handler errors and the latency added before dispatch.

Implement a bounded producer-consumer pipeline with explicit shutdown and error reporting. Keep work outside collection locks and account for memory beyond the input queue.

Implement a stable hash ring, test which keys move, and distinguish balanced ownership from balanced serving traffic.

Implement stable temperature, top-k and top-p sampling with explicit tie rules, masks, invalid inputs and probability tests.

Detect missing and silent ranks from a roster, distinguish liveness from progress, and derive thresholds without assuming independent delays.

Design job creation, cancellation and listing around retries, durable ownership and an explicit pagination contract.

Build a complete segmented metrics-store reference with duplicate-safe range queries, visible retention and tested compaction.

Check useful progress before interpreting GPU activity. Run a bounded trace exercise that exposes a rank-conditional call, equal-count ordering errors and matching shapes with the wrong tensor meaning.

Connect an infrastructure problem you want to solve with work you have actually done. Explain what prompted the move now, what transfers from your background and what you still need to learn.

Clarify what the 256-GPU request must accomplish, then compare it with real placement and quota constraints. Offer a feasible alternative and explain who can decide when teams’ priorities conflict.

Connect your view on AI safety to systems you could build: model access, audit evidence, evaluation capacity and a tested way to stop or reverse a release. Explain the costs and limits of each control.

Explain how a plausible diagnosis became too certain, what disproved it and what changed in your investigation. Separate observed symptoms from causal evidence.

Plan a useful first month by tracing a real job, reviewing incidents, helping under supervision and shipping a bounded improvement with the team.

Handle a request to skip a safety check by identifying the required gate, finding approved alternatives and making exception authority explicit.

Build a learning habit around questions, source evidence and practical tests. Explain a real example without claiming work you did not perform.

Use an evidence-based timeline, specific contributing conditions and verified follow-up to make a postmortem useful.

Support a new responder without turning an outage into an exam. Choose an appropriate role, explain handoffs and review the decisions afterward.

Defend a specific infrastructure opinion with scoped evidence, useful arithmetic and a condition that would change your view.

Explain a decision under uncertainty through the cost of waiting, bounded exposure and evidence that changes the next action.

Connect a researcher tool to the workflow it improved. Measure the whole loop, meaningful adoption and effects on other users.

Offer a first-90-days plan with useful early work and clear decision points. Let the team’s evidence and constraints determine the deliverable.

Explain an isolation finding through the intended boundary, authorized evidence, designated reporting process and checks for both denied and permitted access.

Build a five-year infrastructure outlook from explicit assumptions, competing explanations and evidence that could change your view. Connect each prediction to a practical engineering decision.

Derive resident-block limits and distinguish resident warps from ready work before trading registers for more occupancy.

Trace GPU credits through reservation, allocation, metering and settlement, with crash-safe ownership and a separate fairness policy.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

Trace stable row normalization, handle independent strides and fully masked inputs, then separate fusion traffic from measured speed.

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Use a flexible 45-minute plan to connect requirements, a request path, capacity arithmetic, selected deep dives and a clear closing decision.