serving

192 GB and 5.3 TB/s per card beat the H100 on both memory numbers at a lower hourly price. Whether that wins in production is a question about achieved bandwidth and ROCm kernel coverage, and there is a short list of numbers to run before you sign the order.

Decode throughput is bytes per step over bandwidth, price is dollars per hour, so tokens per dollar for a 70B falls out of a spec sheet in ten lines. In bf16 the three generations are within 10% per dollar; fp8 pulls the H100 ahead, and fp4 on the B200 doubles it again. The chain, the table and the caveats.

Failing at startup and failing under load are different problems with different fixes, and the engine tells you which in its own log. Where the memory actually goes, the four flags that move it, and the arithmetic that says whether it can fit at all.

Expert parallelism reads fewer bytes and pays two all-to-alls, so it loses when the interconnect is wrong or when the routing is uneven. Three candidate causes, the measurement that separates them, and why the third one is the answer more often than the first two.

Both engines cache prefixes, so the choice is not about whether the feature exists. It is about how much of your traffic shares a prefix, what the cache eviction does under pressure, and which tuning model your team can actually operate.

The bound assumes weights stream contiguously and nothing else costs time, and at batch one both assumptions fail badly. Four terms that make up the gap, why sparse models suffer most, and the two fixes that recover most of it.

These two flags decide how much memory the engine reserves and how many users share each step, and both are usually left at values that come from the model rather than from the product. The derivation, the SLO that bounds the second one, and the measurement that confirms both.

Fluent and wrong is a plumbing failure, not a model failure, because a model whose weights are broken produces obvious nonsense. Five places the plumbing goes wrong, the reproduction that isolates each, and the one that is invisible until an evaluation runs.

A single throughput number describes a configuration nobody runs. The four variables that have to be fixed, the curve that is the actual deliverable, and the bandwidth bound that turns the result from a description into a diagnosis.

Halving the weights halves the GPU count and moves the quality evaluation onto you. What the format actually buys in memory and in speed, why those are different questions, and the evaluation that has to run before it ships.

A benchmark that replays one prompt measures the cache and reports a number production will never see. What the hit rate actually depends on, the eviction behaviour that erodes it under load, and the measurement that predicts the real gain.

The two phases want opposite machines, so running them together makes every setting a compromise. What splitting buys, the KV transfer that couples the pools, and the arithmetic that says whether the coupling is affordable for this model.

Forming and then hanging is a different fault from failing to form, and it points at a small set of causes. The rank field that is a cumulative count rather than an index, the collective that never completes, and the three checks that separate configuration from fabric.

Six steps forward from traffic, never backward from an available GPU count. The one input that has to be measured rather than derived, the headroom that is not optional, and the utilization term that moves cost per token more than any tuning flag.

It helps most exactly where these models are weakest, which is single-user latency at low batch, and the published gains are larger than on dense models for a reason. The acceptance arithmetic, what it costs at high concurrency, and the measurement that decides.

A full cache pool with idle streaming multiprocessors is a specific signature with a specific set of remedies. Six of them, ranked by what they cost, and the one that is free because the flag was set from the model rather than the product.

A tail that wide is queueing or prompt-length variance, and the two are distinguishable in one measurement. Four causes ranked by how often they are it, the decomposition that attributes the wait, and the fix that is usually a flag rather than hardware.

Two steps consume most of the week and neither is the one people plan for. The gate that can end the project on day one, the multi-hour download, the correctness check almost nobody runs, and the canary that has to see real traffic before Friday.

The self-hosted price falls with volume and the API price does not, so the two cross at a token rate. Where that crossing sits, the fixed cost floor that makes low volume expensive, and the three reasons that override the arithmetic in both directions.

Identical replicas serving identical traffic should produce identical numbers, so a difference is a defect and the fleet median is the tool that finds it. Five causes, the three that are about the replica and the two that are about what it was sent.

Four things break in order and only one of them is the attention math everyone expects. What a single million-token sequence costs in memory, why the first token takes minutes, and what the model's own architecture already did about the parts that would otherwise be impossible.

A model update is a deploy whose failures are invisible to every deployment metric. What has to be re-validated even for a point release, the canary that watches output shape rather than error rate, and the fingerprint that makes a customer report investigable.

Twenty models of different sizes is a packing problem with a cold-start penalty, and the two pull in opposite directions. What stays resident, what loads on demand, the arithmetic that decides which is which, and the interface that stops every team from asking for a dedicated replica.

Three published numbers turn tuning from guesswork into a loop with a stopping rule. Which flag moves each, the order to adjust them in, and the two out-of-memory failures that have different fixes despite looking the same.

Engine upgrades change defaults, kernel selection and scheduling at once, so attribution is the whole problem. The bisection that works on a version rather than a commit, the four categories of cause, and the pre-upgrade baseline without which none of this is possible.

Most layers stop having a growing cache, which changes capacity planning by a large factor and demands a cache manager most engines did not have. What the memory model becomes, what the engine must implement, and the two features that were disabled by default while it settled.

The cheapest correct answer beats the best answer at scale, so routing is a cost problem with a quality floor. The three routing signals that work, the cascade that trades latency for cost, and the measurement that catches quality loss no latency dashboard shows.

The conversion is the easy part and the evaluation is the project. What calibration data does and why yours should come from production, the per-category evaluation that catches what an average hides, and the parts of the model that should not be quantized at all.

Every constraint in this bank meets in one design and they conflict. What has to be true of the model before the deployment is possible at all, the four mechanisms that make the latency usable, and the honest statement of what the first request still costs.

An image becomes a large number of tokens before the language model sees anything, so a request that looks small carries a prefill the size of a long document. The token arithmetic, the encoder that sits outside the usual parallelism, and the two capacity numbers that move.

Tool calling passes through four layers and a break in any one produces the same symptom, so the debugging is a bisection rather than an investigation. Which layer the size of the drop points at, the deploy diff that usually contains the answer, and why no latency metric moved.

Model evaluations answer whether the weights are good and deployment evaluations answer whether your configuration serves them correctly, which is a different and more common failure. What to sample, why category averages hide the regressions that matter, and the reference that makes a result mean something.

Prompting for JSON produces valid JSON most of the time, and most of the time is not a contract. Constrained decoding makes it a guarantee by restricting what the sampler may choose, and the cost is a compilation step and a throughput effect that depends entirely on how complex the schema is.

One tenant can consume a shared deployment's capacity without exceeding any limit you set, because the limits are usually on requests and the cost is in tokens. What actually needs limiting, the three isolation levels by strength and price, and the fix that works in minutes.

Benchmark rankings answer a question your product did not ask, and candidates usually differ more in what they cost to serve than in what they can do. The four axes in elimination order, and the deployment floor that rules candidates out before quality is discussed.

Graphs remove the per-kernel launch cost, which is a large share of a decode step at low batch and a small one at high batch, so whether it matters is a question about your operating point. The arithmetic, the four reasons an engine turns them off, and the measurement that settles it.

Fifty full copies is fifty deployments and is unaffordable. Fifty low-rank adapters over one shared base is one deployment, because an adapter is a tiny fraction of the weights and adapters from different tenants can share a batch. The arithmetic that makes it work and the two limits that decide how far it goes.

A model reference that names a branch points at whatever is current, so a restart months after a deploy loads whatever the authors published since. What that produces, why the fleet ends up half-updated, and the three changes that make a deployment reproducible.

The weights are portable and almost nothing else is. Four layers have to exist before the model runs at all, the quantization format is the one most likely to be missing, and the honest plan states what will not work in the first quarter rather than promising parity.

The first number every serving interview asks for, derived from parameter count and bytes per parameter, in three precisions, with the part people forget to add.

Two constraints per replica, memory and throughput, each derived from the model and the traffic; whichever binds sets the users per replica, and the replica count follows with headroom. The full chain from 1,000 users to a number of nodes.

Memory follows total parameters and compute follows active ones, so the two models pull the GPU count in opposite directions. The chain for a 235B-total, 22B-active MoE against a dense 70B, for serving and then for training.

Peak sets the fleet, average sets the bill, and the ratio between them is idle money. The chain from a 3x diurnal peak to a replica count, the utilization it implies, what autoscaling can and cannot recover given model load times, and what to do with the trough.

Unlike a training job, a slow serving replica holds nobody else back, so it survives unnoticed while degrading a slice of users. The five causes and the metric that distinguishes each, why routing can create the symptom with no hardware fault, and the fix that limits damage while you investigate.

A replica per variant is 100 copies of a 140 GB base and 800 GPUs mostly idle; multi-LoRA holds the base once and hundreds of adapters in a few gigabytes. The memory arithmetic, the throughput cost of mixed batches, the routing that keeps hot adapters resident, and the two cases where a variant earns its own replica.