observability

Tokens per second flatters a system that is failing its users. The metrics that matter are two latencies at a percentile and the fraction of requests that meet both.

A flat median with a broken tail means one in a hundred requests is hitting something the others do not. There are five usual suspects, and the order you rule them out in is the answer.

The number nvidia-smi calls utilization measures whether any kernel was running, not whether the chip was busy. A decode step that saturates HBM can read as thirty percent. The right dashboard has four other numbers on it.

A training run has one number that matters and a handful that explain it. Goodput as the top-level metric, the per-rank timing that finds a straggler among a thousand, the hardware layer beneath it, and the three dashboards that serve three different people asking three different questions.

GPU utilization at 100% tells you nothing, and a trace per token would cost more than the tokens. The span layout for one request, the GPU counters that read headroom, the cardinality budget that keeps the metrics store alive, and cost attribution that charges tenants for the batch share they used.

Forty teams calling three providers with a hundred API keys and no idea what it costs is what the gateway exists to end. One endpoint with a model alias layer, routing by policy and health, a cache that is safe only for deterministic calls, fallbacks that degrade rather than fail, budgets per team, and the audit trail.