Modal AI Infrastructure Engineer interview questions

The part with no NVLink can be the correct choice, and the reason is the workload shape rather than the specification. What fits on one card, what MIG partitioning buys a multi-tenant service, and the exact point where the missing scale-up link makes the decision flip.

A replica takes a minute to become useful and the traffic changes in seconds. Scale on the wrong signal and you buy GPUs you never use, or you drop requests while they boot. The design is three numbers and two timers.

141 gigabytes have to move from somewhere to eight GPUs, and every hop has a bandwidth. Add the CUDA init, the engine warm-up and the graph capture, and the minute is gone unless you design each step.

The number nvidia-smi calls utilization measures whether any kernel was running, not whether the chip was busy. A decode step that saturates HBM can read as thirty percent. The right dashboard has four other numbers on it.

Failing at startup and failing under load are different problems with different fixes, and the engine tells you which in its own log. Where the memory actually goes, the four flags that move it, and the arithmetic that says whether it can fit at all.

A repository page and a config file contain everything needed to say whether this is a weekend project or a quarter. The nine fields that matter, the two that change the answer by an order of magnitude, and the go or no-go check that comes before any of the arithmetic.

Both engines cache prefixes, so the choice is not about whether the feature exists. It is about how much of your traffic shares a prefix, what the cache eviction does under pressure, and which tuning model your team can actually operate.

These two flags decide how much memory the engine reserves and how many users share each step, and both are usually left at values that come from the model rather than from the product. The derivation, the SLO that bounds the second one, and the measurement that confirms both.

Fluent and wrong is a plumbing failure, not a model failure, because a model whose weights are broken produces obvious nonsense. Five places the plumbing goes wrong, the reproduction that isolates each, and the one that is invisible until an evaluation runs.

A single throughput number describes a configuration nobody runs. The four variables that have to be fixed, the curve that is the actual deliverable, and the bandwidth bound that turns the result from a description into a diagnosis.

A benchmark that replays one prompt measures the cache and reports a number production will never see. What the hit rate actually depends on, the eviction behaviour that erodes it under load, and the measurement that predicts the real gain.

Forming and then hanging is a different fault from failing to form, and it points at a small set of causes. The rank field that is a cumulative count rather than an index, the collective that never completes, and the three checks that separate configuration from fabric.

Six steps forward from traffic, never backward from an available GPU count. The one input that has to be measured rather than derived, the headroom that is not optional, and the utilization term that moves cost per token more than any tuning flag.

A full cache pool with idle streaming multiprocessors is a specific signature with a specific set of remedies. Six of them, ranked by what they cost, and the one that is free because the flag was set from the model rather than the product.

A tail that wide is queueing or prompt-length variance, and the two are distinguishable in one measurement. Four causes ranked by how often they are it, the decomposition that attributes the wait, and the fix that is usually a flag rather than hardware.

Two steps consume most of the week and neither is the one people plan for. The gate that can end the project on day one, the multi-hour download, the correctness check almost nobody runs, and the canary that has to see real traffic before Friday.

The self-hosted price falls with volume and the API price does not, so the two cross at a token rate. Where that crossing sits, the fixed cost floor that makes low volume expensive, and the three reasons that override the arithmetic in both directions.

Identical replicas serving identical traffic should produce identical numbers, so a difference is a defect and the fleet median is the tool that finds it. Five causes, the three that are about the replica and the two that are about what it was sent.

Four things break in order and only one of them is the attention math everyone expects. What a single million-token sequence costs in memory, why the first token takes minutes, and what the model's own architecture already did about the parts that would otherwise be impossible.

A model update is a deploy whose failures are invisible to every deployment metric. What has to be re-validated even for a point release, the canary that watches output shape rather than error rate, and the fingerprint that makes a customer report investigable.

Twenty models of different sizes is a packing problem with a cold-start penalty, and the two pull in opposite directions. What stays resident, what loads on demand, the arithmetic that decides which is which, and the interface that stops every team from asking for a dedicated replica.

Three published numbers turn tuning from guesswork into a loop with a stopping rule. Which flag moves each, the order to adjust them in, and the two out-of-memory failures that have different fixes despite looking the same.

Engine upgrades change defaults, kernel selection and scheduling at once, so attribution is the whole problem. The bisection that works on a version rather than a commit, the four categories of cause, and the pre-upgrade baseline without which none of this is possible.

The cheapest correct answer beats the best answer at scale, so routing is a cost problem with a quality floor. The three routing signals that work, the cascade that trades latency for cost, and the measurement that catches quality loss no latency dashboard shows.

An image becomes a large number of tokens before the language model sees anything, so a request that looks small carries a prefill the size of a long document. The token arithmetic, the encoder that sits outside the usual parallelism, and the two capacity numbers that move.

Tool calling passes through four layers and a break in any one produces the same symptom, so the debugging is a bisection rather than an investigation. Which layer the size of the drop points at, the deploy diff that usually contains the answer, and why no latency metric moved.

Prompting for JSON produces valid JSON most of the time, and most of the time is not a contract. Constrained decoding makes it a guarantee by restricting what the sampler may choose, and the cost is a compilation step and a throughput effect that depends entirely on how complex the schema is.

One tenant can consume a shared deployment's capacity without exceeding any limit you set, because the limits are usually on requests and the cost is in tokens. What actually needs limiting, the three isolation levels by strength and price, and the fix that works in minutes.

Benchmark rankings answer a question your product did not ask, and candidates usually differ more in what they cost to serve than in what they can do. The four axes in elimination order, and the deployment floor that rules candidates out before quality is discussed.

Fifty full copies is fifty deployments and is unaffordable. Fifty low-rank adapters over one shared base is one deployment, because an adapter is a tiny fraction of the weights and adapters from different tenants can share a batch. The arithmetic that makes it work and the two limits that decide how far it goes.

A model reference that names a branch points at whatever is current, so a restart months after a deploy loads whatever the authors published since. What that produces, why the fleet ends up half-updated, and the three changes that make a deployment reproducible.

The four layers between `nvidia.com/gpu: 4` and a container that can run CUDA, the one failure each layer produces, the stranded-GPU arithmetic the integer model causes, and why Dynamic Resource Allocation replaces the count with a claim.

A naive cold start for a 7B function is minutes: pull, start, load, initialize. Under a second means deleting links, not speeding them up: lazy image loading, weights from a local cache, a snapshot of the initialized process with its GPU memory, and a warm pool sized from arrival rate. The chain with a number per link.

Containers share a kernel with the code they run, the wrong boundary for code you did not write. gVisor, Firecracker microVMs and full VMs each buy a stronger one at a cost in memory, start time and GPU access. The overhead arithmetic that turns 50,000 sessions into a host count, and the tiered design.

Four minutes is a 15 GB image pulled through one registry connection; the CUDA error is a container runtime newer than the host driver. The pull arithmetic layer by layer, what a cache hit actually saves, the compatibility rule with its two exceptions, and the image layout that makes both problems go away.

A fleet of ten thousand GPUs generates enough hardware events to page someone hourly, and a rota that receives them stops reading them within a month. The three tests an alert must pass to page, the arithmetic of a sustainable rota, and the automation that has to exist first.

Four mechanics do most of the work: symptom-based rules rather than cause-based, burn rate rather than thresholds, deduplication so one event is one page, and suppression during known windows. The arithmetic for each, and the rule that keeps the set from growing forever.

A serverless GPU is a promise to run a customer's model within seconds of a request while charging only for the seconds it runs. The cold-start chain from image pull to first token, the snapshot that collapses it, the warm pool sized by Little's law, and the isolation and packing decisions that set the margin.

A replica takes 70 seconds to become useful and the burst arrives in 60, so the autoscaler cannot win the first minute; the warm pool has to. The leading signal, the cooldowns derived from the cold start, the warm-pool size from Little's law and the burst profile, and the cost of every second of cold start left in.

An agent thinks for a second, runs a tool for a minute, and repeats a hundred times over an hour. The session state machine and where each piece lives, the sandbox per session with its cold-start budget, the KV cache that should survive the tool call, and checkpoints that resume a step rather than an hour.

Entries differ in size by a factor of fifty, so a count-based cache is meaningless, and evicting the model currently serving requests is a correctness bug rather than a performance one. Size accounting, pinning, and the two failures that must raise rather than silently evict.

The arithmetic is one multiplication and the difficulty is entirely in the events that do not pair cleanly. Clipping to the window, an allocation still running when the period ends, a stop with no start, and why every anomaly must be reported rather than dropped.

A war story with no learning in it scores zero, however dramatic. The five beats of an incident narrative, the decision under uncertainty that is the actual subject of the question, and how to tell it when the incident was somebody else's fault.

Measure the hours before you rank anything, because the task that feels worst is usually not the one that costs most. The ordering rule that keeps automation from causing the outage it was meant to prevent, and why detection always ships before remediation.

Any engineer can cut cost. The question is whether you measured what it cost you elsewhere, and whether you knew that before you shipped it. The three mechanisms that produce most of the savings, and the metric you have to watch on the way down.

The first move is not technical. It is deciding who runs what, because one person serially debugging three incidents is slower than three people in parallel and much slower than one person coordinating. The ordering rule, and the two things that outrank everything else.

How you reported it matters more than how you found it. The internal disclosure that gets a fix instead of a defensive reaction, the four places isolation gaps hide in GPU infrastructure, and the test that keeps the fix from regressing.

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.