25Trace one request through an inference platform. What are the spans, what does each cost, and where does latency actually hide?▼mediumNewOpenAIBaseten4 replies◆ premiumSeven spans from arrival to the last token, with realistic durations for each. The three places latency hides that a naive trace misses entirely, why the decode phase is one span and not four hundred, and the attribute that explains most tail latency in a single field.Open full answer →
19Design observability for an inference fleet: per-request spans, GPU metrics, and cost attribution per tenant.▼mediumNewBasetenOpenAI4 replies○ sign inGPU utilization at 100% tells you nothing, and a trace per token would cost more than the tokens. The span layout for one request, the GPU counters that read headroom, the cardinality budget that keeps the metrics store alive, and cost attribution that charges tenants for the batch share they used.Open full answer →