Core

Rate-Limiting Algorithms

Rate limiters control how much work a caller may submit over time. Token buckets, queues and sliding windows allow different burst and waiting behavior.

This concept is included with a free account. Sign in to continue reading.

free account · no card needed

RELATED CONCEPTS

The GPU Credit Scheduler Pattern Capacity and Backpressure Retry, Backoff and Idempotency Concurrency in Python, Go and C++Inference Platform Architecture The Practical Coding Screen Playbook

PRACTICE THIS IN REAL QUESTIONS

Coding for InfraImplement a token bucket rate limiter. Make it thread-safe, and explain what the two parameters actually control.→AI Infrastructure System DesignDesign rate limiting for an LLM API. Why tokens instead of requests, and how does a bucket work when the cost is unknown until the end?→Coding for InfraImplement sliding-window attention without confusing token positions with ring-buffer slots→CUDA, Triton & Kernel EngineeringGiven the addresses each thread in a warp touched, classify the access pattern: coalesced, strided or random. Write the classifier.→Coding for InfraImplement a bounded request log with timers that remain correct under concurrency→Coding for InfraImplement byte-pair encoding merges, then explain why token counts differ across languages→