17Design the playground behind an LLM API: streaming, sessions, quotas and abuse, for a million free-tier users.▼mediumNewOpenAI4 replies○ sign inThe playground is the API's most public client and its most abused one: free, barely identified, streaming. The transport and its cost per connection, session state that survives a refresh with no GPU holding anything, quotas per user and per fleet, and the abuse controls that stop the free tier becoming a free API.Open full answer →
22Design the runtime for long-running agents: sessions, tool sandboxes, KV that lives for hours, and checkpoints.▼hardNewAnthropicOpenAIModal4 replies◆ premiumAn agent thinks for a second, runs a tool for a minute, and repeats a hundred times over an hour. The session state machine and where each piece lives, the sandbox per session with its cold-start budget, the KV cache that should survive the tool call, and checkpoints that resume a step rather than an hour.Open full answer →