29Design an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.▼mediumNewDatabricksSnowflake4 replies◆ premiumForty teams calling three providers with a hundred API keys and no idea what it costs is what the gateway exists to end. One endpoint with a model alias layer, routing by policy and health, a cache that is safe only for deterministic calls, fallbacks that degrade rather than fail, budgets per team, and the audit trail.Open full answer →
04Implement retry with exponential backoff and jitter. Why is the jitter the part that matters, and what must never be retried?▼easyNewOpenAIBaseten4 repliesunlockedExponential backoff spaces one client's retries and does nothing for a thousand clients that failed together, which is the case that matters. The measured difference jitter makes to the peak arrival rate, the two budgets that bound the damage, and the classification of what is safe to retry at all.Open full answer →
