AI Infra Interviews logo
← AI Infrastructure System Design / 15
mediumOpenAIAnthropic

Design rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?

Divide one token budget across gateways instead of copying it. Follow output reservation and settlement, then handle grants, retries and outages without inventing extra burst capacity.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Divide one token budget across gateways instead of copying it. Follow output reservation and settlement, then handle grants, retries and outages without inventing extra burst capacity.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core
💻 Coding for InfraSign in
Rate-Limiting AlgorithmsRate limiters control how much work a caller may submit over time. Token buckets, queues and sliding windows allow different burst and waiting behavior.
Advanced
📐 AI Systems Design🔒 Premium
Capacity and BackpressureBackpressure limits accepted work when a service cannot keep up. Bounded queues and clear retry rules prevent overload from spreading through the request path.
Core
📐 AI Systems DesignSign in
GPU Job Scheduler DesignA GPU scheduler decides which jobs run, where they run and when resources can be reclaimed. Its placement rules must account for whole jobs, topology and tenant policy.
Core
📐 AI Systems DesignSign in
Designing for Latency SLOsA latency budget follows a request through networking, queueing and model execution. Use measurements to find which stage causes slow responses under load.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackDesign a multi-region inference deployment: capacity per region, routing, failover, and getting the weights everywhere.Next in this trackDesign the playground behind an LLM API: streaming, sessions, quotas and abuse, for a million free-tier users.Next in this trackDesign the checkpoint store for a lab running several large training jobs: write bursts, retention, resharding and lineage.
FEDITOR'S NOTE

Scored on explicit capacity assumptions, correct state and identity boundaries, failure handling and a justified design choice.

DISCUSSION · 0

No comments yet — be the first to share your approach.