coding
AI infra interview questions tagged coding, across every topic.
2 questions · 2 unlocked for you
Concepts behind "coding"
The curriculum that explains the ideas these questions test.
Foundational
The GPU Credit Scheduler PatternThe most widely reported coding problem in AI infrastructure loops is a small scheduler: accounts hold credits, jobs arrive with a cost and a priority, and you must decide which jobs run, in what order, without letting any account overspend, then extend it under follow-ups (refunds, reservations, concurrency limits, fairness). It is not a trick question; it is a test of whether you can model state cleanly, pick the right data structures, keep invariants under mutation, and talk about complexity while typing. This page works the problem from the first line to the fourth follow-up, with the code, the invariants, and the derivations.💻 Coding for Infra
Foundational
Cache-Friendly Data StructuresA cache line is 64 bytes and it is the unit of coherence, so where data sits decides how fast code runs more often than which algorithm it uses. Two consequences dominate infrastructure code: a lookup that chases a pointer pays two dependent memory stalls instead of one, and two threads updating adjacent variables contend for a line they do not logically share. Both are layout problems with layout fixes.💻 Coding for Infra
Foundational
Consistent Hashing and ShardingSplitting work across N servers with a modulo of N moves almost everything when N changes, which for a cache means throwing away almost all of it. Consistent hashing places servers and keys on a ring so adding or removing one moves only its share, and virtual nodes fix the imbalance a small ring otherwise has. In LLM serving the same structure routes requests by prompt prefix so a conversation reaches the replica already holding its cache.💻 Coding for Infra
Core
Rate-Limiting AlgorithmsA rate limiter answers one question, 'may this request proceed now?', and the three classic algorithms answer it with different shapes of fairness and memory: the token bucket allows bursts up to a capacity and refills at a rate, the leaky bucket smooths output to a fixed rate, and sliding windows count recent requests exactly or approximately. AI platforms limit in tokens as well as requests, per tenant, across many gateways, which adds two twists: a request's cost is unknown until it finishes, and the counters must be shared. This page derives each algorithm, implements the token bucket correctly, and covers both twists.💻 Coding for InfraSign in
Core
Retry, Backoff and IdempotencyA retry is a second request that the system did not budget for, and a thousand clients retrying at the same moment is a second outage that the first one caused. The craft is small and specific: retry only what is safe to retry, wait an exponentially growing random interval so the retries spread out, cap the total retries with a budget, and make every retried operation idempotent so a duplicate does not double-charge or double-train. This page derives why synchronized retries double the load, works the jitter arithmetic, implements the client correctly, and covers idempotency keys for the operations an AI platform exposes.💻 Coding for InfraSign in
Advanced
Batching Queues and BackpressureWrite a request batcher is the coding round's version of the serving engine's scheduler: requests arrive one at a time, the GPU wants them in groups, and the batcher decides when a group is full enough to send without holding anyone too long or accepting more than it can hold. The two knobs are the maximum batch size and the maximum wait, the invariant is a bounded queue, and the follow-ups (priorities, cost-aware batching, cancellation, bounded in-flight batches) are the ideas the real engines carry. This page implements the batcher in asyncio, derives what each knob buys, and walks the follow-ups.💻 Coding for Infra🔒 Premium