Anyscale AI Infrastructure Engineer interview questions

Treat recovery as a new worker generation, then compare spare replacement with a legal shrink and its batch consequences.

One request finishes, another keeps writing, and a third waits outside. Follow the same workload through two schedules to see what continuous batching improves, and what it cannot promise.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Follow adapter-specific token rows through a shared base, calculate stored adapter bytes, and separate the resident set from the active batch.

Forecast demand across startup delay, track replicas already being created, and drain slowly enough to avoid repeating the same scale decision.

Separate device throughput bounds from measured service curves, then choose a feasible operating point using latency, quality and goodput.

A measured run shows where the time goes in a small batch job, then why every result parsed but none was deliverable. The answer turns on startup cost, co-tenant latency and a per-item contract.

Separate frozen weights from trainable state, count rank-16 adapters from matrix shapes, and check peak memory before choosing a fine-tuning layout.

Use Little’s law to distinguish waiting and active requests, then test saturation with a stated service model. Avoid turning a mean occupancy identity into a universal latency rule.

Choose the control plane from job admission, placement, recovery and service needs, with a concrete partial-allocation failure.

See how partial allocation can strand two training jobs. Separate group admission, real placement, readiness and recovery instead of assuming atomic pod startup.

Balance durable checkpoint cost against replay and restart loss, with explicit assumptions and a warning sign when the approximation breaks down.

Build a durable, bounded scheduling loop with reservations and checkpoint-aware preemption, and avoid universal queue-wait formulas.

Trace Kubernetes pod placement and Ray bundle reservation as separate layers, including logical GPU limits and autoscaling delay.

Choose the tenant boundary before planning base-model reuse. Account for frozen weights, adapter state and activations separately, then schedule by compatibility, fairness and the real cost of restarting training.

Calculate traffic across the actual inter-cloud boundary before deciding. Whole-job placement often avoids repeated transfers, but communication volume, overlap, convergence requirements and available links determine whether a split job is viable.

Trace the job from quota reservation through admission to physical placement. Count eligible node shapes, not just free GPUs, and use controller and scheduler evidence to choose the least disruptive fix.

Follow preemption through checkpoint success or timeout, a safe stop barrier and requeue. See why an expired lease is not enough to free a GPU, and how observed checkpoint age changes victim cost.

Derive adapter bytes from the target matrices, share compatible base weights, and keep each request's cache identity separate. A worked pool design avoids applying the mixed-adapter cost twice.

Implement priority admission and completion, preserve a preempted job’s sequence, and distinguish a sufficient greedy victim set from an optimal one.

Simulate admission, decode growth and preemption. Check memory accounting and useful progress with the same finite workload.

Implement owner-pop and thief-steal semantics with correct accounting for running tasks that generate children.

Follow one request from API admission to streamed output, then zoom into the token-generation loop. See where the cache lives, who schedules work, and how workload measurements turn the drawing into a capacity plan.

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Trace GPU credits through reservation, allocation, metering and settlement, with crash-safe ownership and a separate fairness policy.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.

Derive a 10,240-GPU layout from model work, then account for collective traffic, durable checkpoints, power and replacements. A nested diagram connects tensor, pipeline and data parallel groups to the fleet.

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Separate quota, borrowing, reclaim latency and fairness, with a demand-conserving example of when pooling improves utilization.