Anyscale AI Infrastructure Engineer interview questions

A dead node at 4 a.m. costs twenty minutes if a human restarts it and under five if the system does. The four stages of an automatic recovery, the reason a lost node takes a whole pipeline replica with it, why hot spares beat resharding, and the minutes that each stage still costs even when everything works.

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

Three hundred fine-tunes as three hundred deployments would need three hundred GPUs. Batched adapters put them on one base, and the arithmetic says what each extra adapter costs in memory and in step time.

A replica takes a minute to become useful and the traffic changes in seconds. Scale on the wrong signal and you buy GPUs you never use, or you drop requests while they boot. The design is three numbers and two timers.

Tokens per second climbs with batch and per-token latency climbs with it. The curve has a knee set by the hardware ridge, and the operating point is where your latency SLO cuts it.

Sixteen bytes per parameter is 1.1 TB of static state before a single activation; LoRA keeps the 141 GB of frozen weights and shrinks the rest to a few gigabytes. The derivation, the GPU counts, and the activations that both still pay.

Concurrency equals arrival rate times residence time, and residence time on an LLM replica grows with concurrency. The chain from 50 requests a second to 1,000 in flight, the replica count with a stated utilization, and the feedback loop that turns 90% utilization into a queue that does not drain.

Slurm gives you gang scheduling, topology and MPI launch for free and nothing for services; Kubernetes gives you services, elasticity and the ecosystem and makes you bolt on the batch semantics. The hybrid that most large clusters land on, with the numbers that push them there.

A distributed training job is 64 pods that start together or not at all. The deadlock two partially placed jobs produce, counted out on a 64-GPU cluster, and how Kueue and Volcano make the job the unit of admission.

Spot is 60% cheaper until the preemptions eat the discount. The expected-loss formula (waste per preemption = half the checkpoint interval plus restart), the break-even worked at three cadences, the checkpoint cost that pushes back the other way, and the gang arithmetic that says which jobs must never be on spot.

A 100,000-entry queue cannot be scanned every cycle, a big gang cannot wait forever behind small jobs, and a preemption must not discard an hour of work. The loop in order (admit, order, place, reserve, backfill, preempt), the queue-wait arithmetic, and the checkpoint-aware victim rule.

Ray gives you actors, tasks and placement groups inside a cluster of pods that Kubernetes thinks are just pods. Two schedulers, two views of the same GPUs: the idle-node arithmetic, the placement group that cannot be gang-scheduled by the layer below, and the rules that stop the two from wasting each other's capacity.

The expensive thing is a base model in GPU memory, not the customer's job. Keep bases resident, pack LoRA jobs onto them, isolate tenants at the process and device level, checkpoint adapters in seconds so jobs run on cheap capacity, and attribute cost per tenant. The arithmetic behind each rule.

A synchronous training job across two clouds runs at the speed of the link between them, a thousand times slower than the fabric inside either. The arithmetic that kills the idea for one job, the two cases where it is tolerable, and what a platform does instead: partition the work, not the job, and move data ahead.

Free GPUs and a pending gang are not a contradiction: the gang needs 64 whole nodes at once and the free GPUs sit on 300 half-used nodes. The isolation order (quota, fit, per-node reasons, one node's device plugin), the command at each step and what it means, and the fixes ranked by displacement.

A hundred thousand jobs is a queue you cannot scan per tick, a preemption policy that has to know when each victim last checkpointed, and a state machine with one transition that most designs get wrong. The data model, the tick loop, the checkpoint-aware eviction and the arithmetic behind each.

A replica per variant is 100 copies of a 140 GB base and 800 GPUs mostly idle; multi-LoRA holds the base once and hundreds of adapters in a few gigabytes. The memory arithmetic, the throughput cost of mixed batches, the routing that keeps hot adapters resident, and the two cases where a variant earns its own replica.

A heap orders the queue and the interesting logic is elsewhere: which running jobs may be evicted, how many, and what happens to them. The victim-selection rule that avoids evicting more than necessary, the equal-priority case that must not preempt, and the starvation the design creates if nothing ages.

Admission, a step that grows every sequence, and preemption under memory pressure. The loop is short and it has a failure that only appears when the cache is tight: in a measured run it preempted seventeen times and completed nothing, because every preempted sequence restarted from zero.

Two ends, two access patterns, and one design choice that removes almost all contention. Why the owner takes the newest task and thieves take the oldest, what that does to cache locality, and the measurement showing why this cannot be demonstrated in Python at all.

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.

Weight-only int4, fp8 weights and activations, or fp8 for the cache alone: each fixes a different bottleneck. Match the method to the resource you are short of, then run the evals that catch what perplexity misses.