caching

Every constraint in this bank meets in one design and they conflict. What has to be true of the model before the deployment is possible at all, the four mechanisms that make the latency usable, and the honest statement of what the first request still costs.

Forty teams calling three providers with a hundred API keys and no idea what it costs is what the gateway exists to end. One endpoint with a model alias layer, routing by policy and health, a cache that is safe only for deterministic calls, fallbacks that degrade rather than fail, budgets per team, and the audit trail.

Entries differ in size by a factor of fifty, so a count-based cache is meaningless, and evicting the model currently serving requests is a correctness bug rather than a performance one. Size accounting, pinning, and the two failures that must raise rather than silently evict.