mediumNewOpenAIAnthropic

Design rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?

A request can cost 50 tokens or 50,000, so a request limit protects nothing. The two buckets per tenant per model, the reservation-then-settle scheme for output tokens you cannot count in advance, the distributed counter fast enough for the gateway, and how the limits map onto the fleet's real capacity.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

💻 Coding for InfraSign in

Rate-Limiting AlgorithmsA rate limiter answers one question, 'may this request proceed now?', and the three classic algorithms answer it with different shapes of fairness and memory: the token bucket allows bursts up to a capacity and refills at a rate, the leaky bucket smooths output to a fixed rate, and sliding windows count recent requests exactly or approximately. AI platforms limit in tokens as well as requests, per tenant, across many gateways, which adds two twists: a request's cost is unknown until it finishes, and the counters must be shared. This page derives each algorithm, implements the token bucket correctly, and covers both twists.

Advanced

📐 AI Systems Design🔒 Premium

Capacity and BackpressureA system that accepts more work than it can finish does not degrade gracefully; it degrades completely, because every request it queues makes every other request slower until all of them time out. The defence is backpressure: bounded queues at every stage, admission control that rejects early when the expected wait exceeds the budget, load shedding by priority, and clients that back off. For LLM serving the stages are the gateway, the router, the engine's queue and its KV pool. This page works the arithmetic of why unbounded queues kill p99, designs the bounds per stage, and the client contract that keeps it stable under overload.

Core

📐 AI Systems DesignSign in

GPU Job Scheduler DesignDesign a scheduler for a shared GPU cluster is the most common design prompt in AI infrastructure interviews, because it touches everything: queues and priorities, gang placement, topology, fairness across teams, preemption and the checkpoints that make it survivable, and the failure handling that keeps a 512-GPU job alive. This page builds the design in layers, states the data model and the scheduling loop, derives the numbers (how long a job waits, how much preemption costs, how much fragmentation wastes), and lists the trade-offs the interviewer will push on.

Core

📐 AI Systems DesignSign in

Designing for Latency SLOsA latency objective is met or missed by the sum of a chain of delays, and the way to design for it is to write the chain down with a number on every link, find the links that dominate at the tail, and attack those. For an LLM request the chain is network, gateway, router, queue, prefill, then the decode loop, and the tail is shaped by queueing and by the size of the batch the request lands in. This page decomposes a 500 ms time-to-first-token budget link by link, derives how queueing turns a comfortable median into a broken p99, and gives the design moves (admission control, chunked prefill, priority lanes, hedging) that hold it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign a multi-region inference deployment: capacity per region, routing, failover, and getting the weights everywhere.Next in this trackDesign the playground behind an LLM API: streaming, sessions, quotas and abuse, for a million free-tier users.Next in this trackDesign the checkpoint store for a lab running several large training jobs: write bursts, retention, resharding and lineage.

FEDITOR'S NOTE

Scored on limiting tokens and requests separately, on reserving estimated output tokens and settling on completion, on a gateway-local counter with periodic sync rather than a synchronous central store, and on deriving limits from fleet capacity.

DISCUSSION · 0

No comments yet — be the first to share your approach.