Together AI AI Infrastructure Engineer interview questions

Peak FLOPS divided by bandwidth is one number per chip, and it decides whether any kernel can ever reach peak. How to compute it, how to compute a kernel's intensity from its bytes and FLOPs, and how to read the answer before writing a line of CUDA.

At batch 1 a decode step reads every weight byte to do two FLOPs with it. The intensity is the batch size, the ridge is about 295, and the KV cache puts a ceiling on how far batching can push you. The full chain for a 70B model on H100.

Four percentages from a profiler summary are enough to place a kernel on the roofline, estimate its arithmetic intensity, and rule out half the optimizations people reach for. The reading, the arithmetic that backs it, and the fix list in the order it pays.

A linear layer at batch 1 runs at a third of a percent of peak and at batch 4,096 near 70%. The curve between has two regions and a sawtooth: the memory-bound slope where intensity equals M, the plateau where the tensor cores are the limit, and the dips where the tile count does not divide the SM count.

Decode throughput is bytes per step over bandwidth, price is dollars per hour, so tokens per dollar for a 70B falls out of a spec sheet in ten lines. In bf16 the three generations are within 10% per dollar; fp8 pulls the H100 ahead, and fp4 on the B200 doubles it again. The chain, the table and the caveats.

Decode is bandwidth-bound, so the FP8 FLOPS number that dominates the marketing does not move it. Where the B300 pays is capacity, and capacity converts into throughput through batch size rather than directly. The arithmetic that decides it, and the case where the B200 wins.

Shared memory has 32 banks, each 4 bytes wide, and a warp's access is as slow as the most-loaded bank. The bank of an address, why a column walk down a 32-wide tile puts all 32 lanes in one bank, the padding by one column that spreads them across all 32, and the profiler counter that confirms the fix.

A chain of five elementwise operations reads and writes the tensor five times when once would do. The byte arithmetic for an unfused chain against a fused one, the launch overhead that matters at small sizes, the three fusion shapes, and the operations where fusion changes nothing because the matmul was at the roof.

One program per row, the row in registers, max then exp then sum then divide, one read and one write of HBM. The runnable kernel with its launch, the numerics (subtract the max, accumulate in fp32), the block-size rule and the wide-row limit, and the arithmetic that says the kernel is done at 90% of copy speed.

Standard attention writes an N by N score matrix to HBM and reads it back twice. At 8k context that is 134 MB per head. The traffic arithmetic before and after tiling, the online-softmax identity that makes one pass possible, where the kernel lands on the roofline, and why the memory saving matters more than the speed.

Each version fixed a different bottleneck: version 1 fixed memory traffic, version 2 fixed non-matmul work and parallelism, version 3 fixed the fact that softmax and matmul were waiting for each other on Hopper. The arithmetic that shows why a softmax costing one percent of the FLOPs can cost sixty percent of the time.

Triton hands you a block of data and writes the thread-level code for you, which covers most of what a serving or training stack actually needs. The four things it does not give you, the performance you give up in each case with numbers, the reversal as Triton gains Hopper features, and how to decide in the room.

At batch 1 a decode step is a few milliseconds of GPU work behind several hundred kernel launches, and the CPU cannot issue them fast enough. The arithmetic showing the GPU starve, what replay changes, and the four constraints capture imposes on memory.

The KV for one sequence is scattered across fixed-size pages, so the kernel reads a table of physical block numbers before it can read any keys. What the indirection costs, what the table costs in memory, why decode must split the KV dimension across thread blocks, and how the partial softmax states combine.

The target runs one forward pass over all k drafted tokens, compares its distribution against the draft's at each position, and keeps the longest prefix that survives a coin flip. The acceptance rule that makes the output exactly a target sample, the resampling at the first rejection, and the one-integer rollback.

Routing hands each expert a different number of tokens, so the layer is a list of ragged matmuls rather than one. Three separate costs a loop pays, the measured share of compute that skewed routing wastes on tile padding, and what a single grouped launch fixes and what it cannot.

A bug that scales with the number of tiles is invisible in a unit test that fits in one tile. The three candidates that produce exactly this signature, the measurement that separates them, and a test design that would have caught all three before the kernel shipped.

LoRA shrinks the trainable state and leaves 141 GB of frozen weights untouched. QLoRA attacks the frozen half: NF4 takes it to 35 GB, double quantization compresses the scales themselves, paged optimizers absorb the spikes. The byte accounting, what dequantization costs, and where DoRA adds a term.

One forward pass reads every weight. Whether that read is the bottleneck depends on how many tokens ride along with it, and the answer is different for the two halves of a request.

A batch of eight requests finishes when the longest one does, and seven slots compute padding until then. The fix is to make the scheduling unit one decode step, and the throughput math shows why that was worth several times the hardware.

Before vLLM, engines reserved the maximum context for every request up front and wasted most of it. The fix borrowed virtual memory from the operating system, and the numbers show how much capacity it returned.

Weights take a fixed slice, the KV cache takes the rest, and the rest divided by context is your user count. Work it in bf16 and fp8, then find out why the answer also depends on how you split the model.

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.

Weight-only int4, fp8 weights and activations, or fp8 for the cache alone: each fixes a different bottleneck. Match the method to the resource you are short of, then run the evals that catch what perplexity misses.

Three engines, one hardware roofline, and the difference between them is which part of the roofline each one reaches first on your workload. The decision is a table, and the table has reversal conditions.

671 billion parameters that touch 37 billion per token: the memory bill is for the big number and the bandwidth bill is for the small one, until batch and routing collide. The cache is tiny, and that changes the whole fit.

All three guess tokens for the big model to check. They differ in where the guess comes from, how many candidates they verify per step, and how often the target agrees. Acceptance length per unit of draft cost is the number that decides.

Tokens per second climbs with batch and per-token latency climbs with it. The curve has a knee set by the hardware ridge, and the operating point is where your latency SLO cuts it.

Ten thousand requests a second is a fleet of hundreds of nodes, a router that has to know what every replica is caching, and an SLO pair that decides the batch on every one of them. Here is the sizing chain, node by node.

The answer comes from the model card and config.json in five minutes, before downloading anything. The weight footprint, the KV per token that is far smaller than the parameter count suggests, and the parallel degree that has to divide the head count.

Failing at startup and failing under load are different problems with different fixes, and the engine tells you which in its own log. Where the memory actually goes, the four flags that move it, and the arithmetic that says whether it can fit at all.

A repository page and a config file contain everything needed to say whether this is a weekend project or a quarter. The nine fields that matter, the two that change the answer by an order of magnitude, and the go or no-go check that comes before any of the arithmetic.

Expert parallelism reads fewer bytes and pays two all-to-alls, so it loses when the interconnect is wrong or when the routing is uneven. Three candidate causes, the measurement that separates them, and why the third one is the answer more often than the first two.

Both engines cache prefixes, so the choice is not about whether the feature exists. It is about how much of your traffic shares a prefix, what the cache eviction does under pressure, and which tuning model your team can actually operate.

Two point eight trillion parameters fits in one node, because the experts ship in a four-bit format. The footprint arithmetic that reproduces the project's own published hardware minimum, the hybrid cache that needs specific engine support, and why batch-one throughput is a tenth of the bound.

The bound assumes weights stream contiguously and nothing else costs time, and at batch one both assumptions fail badly. Four terms that make up the gap, why sparse models suffer most, and the two fixes that recover most of it.

These two flags decide how much memory the engine reserves and how many users share each step, and both are usually left at values that come from the model rather than from the product. The derivation, the SLO that bounds the second one, and the measurement that confirms both.

Fluent and wrong is a plumbing failure, not a model failure, because a model whose weights are broken produces obvious nonsense. Five places the plumbing goes wrong, the reproduction that isolates each, and the one that is invisible until an evaluation runs.

A single throughput number describes a configuration nobody runs. The four variables that have to be fixed, the curve that is the actual deliverable, and the bandwidth bound that turns the result from a description into a diagnosis.

Halving the weights halves the GPU count and moves the quality evaluation onto you. What the format actually buys in memory and in speed, why those are different questions, and the evaluation that has to run before it ships.

A benchmark that replays one prompt measures the cache and reports a number production will never see. What the hit rate actually depends on, the eviction behaviour that erodes it under load, and the measurement that predicts the real gain.

The two phases want opposite machines, so running them together makes every setting a compromise. What splitting buys, the KV transfer that couples the pools, and the arithmetic that says whether the coupling is affordable for this model.

Forming and then hanging is a different fault from failing to form, and it points at a small set of causes. The rank field that is a cumulative count rather than an index, the collective that never completes, and the three checks that separate configuration from fabric.

Six steps forward from traffic, never backward from an available GPU count. The one input that has to be measured rather than derived, the headroom that is not optional, and the utilization term that moves cost per token more than any tuning flag.

It helps most exactly where these models are weakest, which is single-user latency at low batch, and the published gains are larger than on dense models for a reason. The acceptance arithmetic, what it costs at high concurrency, and the measurement that decides.

A full cache pool with idle streaming multiprocessors is a specific signature with a specific set of remedies. Six of them, ranked by what they cost, and the one that is free because the flag was set from the model rather than the product.

A tail that wide is queueing or prompt-length variance, and the two are distinguishable in one measurement. Four causes ranked by how often they are it, the decomposition that attributes the wait, and the fix that is usually a flag rather than hardware.

Two steps consume most of the week and neither is the one people plan for. The gate that can end the project on day one, the multi-hour download, the correctness check almost nobody runs, and the canary that has to see real traffic before Friday.

The self-hosted price falls with volume and the API price does not, so the two cross at a token rate. Where that crossing sits, the fixed cost floor that makes low volume expensive, and the three reasons that override the arithmetic in both directions.

Identical replicas serving identical traffic should produce identical numbers, so a difference is a defect and the fleet median is the tool that finds it. Five causes, the three that are about the replica and the two that are about what it was sent.

Four things break in order and only one of them is the attention math everyone expects. What a single million-token sequence costs in memory, why the first token takes minutes, and what the model's own architecture already did about the parts that would otherwise be impossible.

A model update is a deploy whose failures are invisible to every deployment metric. What has to be re-validated even for a point release, the canary that watches output shape rather than error rate, and the fingerprint that makes a customer report investigable.

Twenty models of different sizes is a packing problem with a cold-start penalty, and the two pull in opposite directions. What stays resident, what loads on demand, the arithmetic that decides which is which, and the interface that stops every team from asking for a dedicated replica.

Three published numbers turn tuning from guesswork into a loop with a stopping rule. Which flag moves each, the order to adjust them in, and the two out-of-memory failures that have different fixes despite looking the same.

Engine upgrades change defaults, kernel selection and scheduling at once, so attribution is the whole problem. The bisection that works on a version rather than a commit, the four categories of cause, and the pre-upgrade baseline without which none of this is possible.

Most layers stop having a growing cache, which changes capacity planning by a large factor and demands a cache manager most engines did not have. What the memory model becomes, what the engine must implement, and the two features that were disabled by default while it settled.

The cheapest correct answer beats the best answer at scale, so routing is a cost problem with a quality floor. The three routing signals that work, the cascade that trades latency for cost, and the measurement that catches quality loss no latency dashboard shows.

The conversion is the easy part and the evaluation is the project. What calibration data does and why yours should come from production, the per-category evaluation that catches what an average hides, and the parts of the model that should not be quantized at all.

Every constraint in this bank meets in one design and they conflict. What has to be true of the model before the deployment is possible at all, the four mechanisms that make the latency usable, and the honest statement of what the first request still costs.

An image becomes a large number of tokens before the language model sees anything, so a request that looks small carries a prefill the size of a long document. The token arithmetic, the encoder that sits outside the usual parallelism, and the two capacity numbers that move.

Tool calling passes through four layers and a break in any one produces the same symptom, so the debugging is a bisection rather than an investigation. Which layer the size of the drop points at, the deploy diff that usually contains the answer, and why no latency metric moved.

Model evaluations answer whether the weights are good and deployment evaluations answer whether your configuration serves them correctly, which is a different and more common failure. What to sample, why category averages hide the regressions that matter, and the reference that makes a result mean something.

Prompting for JSON produces valid JSON most of the time, and most of the time is not a contract. Constrained decoding makes it a guarantee by restricting what the sampler may choose, and the cost is a compilation step and a throughput effect that depends entirely on how complex the schema is.

One tenant can consume a shared deployment's capacity without exceeding any limit you set, because the limits are usually on requests and the cost is in tokens. What actually needs limiting, the three isolation levels by strength and price, and the fix that works in minutes.

Benchmark rankings answer a question your product did not ask, and candidates usually differ more in what they cost to serve than in what they can do. The four axes in elimination order, and the deployment floor that rules candidates out before quality is discussed.

Graphs remove the per-kernel launch cost, which is a large share of a decode step at low batch and a small one at high batch, so whether it matters is a question about your operating point. The arithmetic, the four reasons an engine turns them off, and the measurement that settles it.

Fifty full copies is fifty deployments and is unaffordable. Fifty low-rank adapters over one shared base is one deployment, because an adapter is a tiny fraction of the weights and adapters from different tenants can share a batch. The arithmetic that makes it work and the two limits that decide how far it goes.

A model reference that names a branch points at whatever is current, so a restart months after a deploy loads whatever the authors published since. What that produces, why the fleet ends up half-updated, and the three changes that make a deployment reproducible.

The weights are portable and almost nothing else is. Four layers have to exist before the model runs at all, the quantization format is the one most likely to be missing, and the honest plan states what will not work in the first quarter rather than promising parity.

Why decode speed is a division of bandwidth by weight bytes, why the answer is about 24 tokens a second regardless of how fast the tensor cores are, and the two ways to double it.

Fleet dollars per second over tokens per second, times a million: a worked $2.78, the utilization term that turns it into $6.94, and why batch is the lever that moves it by 50x.

Two constraints per replica, memory and throughput, each derived from the model and the traffic; whichever binds sets the users per replica, and the replica count follows with headroom. The full chain from 1,000 users to a number of nodes.

fp8 halves the bytes per decode step and doubles the compute peak, but the all-reduces and launch overhead do not shrink, so the measured gain is 1.5x at a fixed batch and about 2x once the freed memory raises the batch. The chain, with the step-time table.

For bf16 weights the decode intensity equals the batch, so the ridge is at batch 295 on an H100. Then the KV cache enters and the answer changes shape: the projections go compute-bound at 300 while the attention reads never do, and at 8k context the step as a whole stays memory-bound at any batch.

Decode is bandwidth-bound, so cost per token tracks dollars per terabyte per second, and on spec the MI300X wins by 2x. The table, the memory-capacity term that changes the replica shape, and the software-efficiency discount that decides whether the spec ranking survives a benchmark.

Two bounds, bandwidth and compute, and five questions about what the number counts. For a 70B on an H100 the claim exceeds the bf16 compute peak and is impossible; for an 8B at high batch it is routine; for input tokens it is easy. The chain that tells the cases apart.

It depends entirely on which dimension you split the model along, and the three choices differ by two orders of magnitude in what they demand. The per-token traffic for each, the arithmetic that puts tensor parallelism on the fast link and pipeline parallelism anywhere, and where disaggregated serving lands.

Fractions are either hardware slices with fixed shapes or soft shares with no isolation, and the scheduler has to know which. The representation, the packing score, the fragmentation arithmetic that spread placement produces, and the defragmentation step that fixes it.

Allocated is not usable: a cluster full of half-empty nodes has hundreds of idle GPUs that no gang can take. The arithmetic of how mixed job sizes fragment a fleet, the packing policies that prevent it, the defragmentation moves that repair it, and what each costs the jobs already running.

The expensive thing is a base model in GPU memory, not the customer's job. Keep bases resident, pack LoRA jobs onto them, isolate tenants at the process and device level, checkpoint adapters in seconds so jobs run on cheap capacity, and attribute cost per tenant. The arithmetic behind each rule.

Unlike a training job, a slow serving replica holds nobody else back, so it survives unnoticed while degrading a slice of users. The five causes and the metric that distinguishes each, why routing can create the symptom with no hardware fault, and the fix that limits damage while you investigate.

Speculative decoding buys latency with spare compute, and a busy fleet has no spare compute. The speedup formula with acceptance rate and draft cost inside it, the batch size where the gain turns negative, where the draft runs and what it costs, the monitoring that catches a silent regression, and rollout per class.

Billing is the one pipeline where a dropped event is money and a duplicated one is a support ticket. Where tokens are counted and by whom, the usage event and its idempotency key, the aggregation that closes a month, the reconciliation that proves the ledger against the engines, and the six ways it goes wrong.

Two triggers, one shared queue, and a timer task that will deadlock the whole thing if it cancels itself. The implementation with the bug I hit and its fix, the latency the window costs, and the error path that decides whether one bad batch fails one caller or all of them.

Three one-line policy differences produce different fragmentation, and the metric that separates them is not utilization. The measured outcome on the same job sequence, why the largest free block is what matters on a training cluster, and the case where the intuitive policy is exactly wrong.

A free list, a per-sequence block table, and a reference count are the whole structure. What forking costs, why the first write after a fork must copy, and the measured waste that this design accepts in exchange for eliminating the far larger waste of pre-allocation.

A geometric process with a cap, a closed form that a twenty-line simulation confirms, and a speedup that turns negative when the verification stops being free. The table across acceptance rates and draft lengths, and the crossover from a 2.35 times win to a 0.75 times loss under two stated cost regimes.

A speedup number with no method behind it is worth nothing, and experienced interviewers stop listening at the number and start asking about the baseline. What makes a performance claim survive scrutiny, and the four ways a real one turns out to be smaller than it looked.

Any engineer can cut cost. The question is whether you measured what it cost you elsewhere, and whether you knew that before you shipped it. The three mechanisms that produce most of the savings, and the metric you have to watch on the way down.

Every token a model has seen leaves a key and a value in every layer. Multiply that out for a 70B model and you will see why memory, not compute, caps how many users a GPU can hold.

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

One 20k-token prompt can freeze every active stream for most of a second. The fix slices it into per-step budgets, and the budget number is a trade between two SLOs you can compute.

Same model size, three attention layouts, a 30x spread in cache per token. The formulas are short, and they explain why DeepSeek can hold four times the context of a dense 70B on the same cards.

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

The same 2,000 tokens are prefilled a thousand times an hour. Caching their KV by content turns that into one prefill and a table lookup, and a radix tree is what makes multi-turn and branching agents share it too.

Sequential is fifty seconds; one batch is four. In between are the questions the interviewer is holding: how you pack ragged inputs, what a batch window costs, and where the memory stops you.

A flat median with a broken tail means one in a hundred requests is hitting something the others do not. There are five usual suspects, and the order you rule them out in is the answer.