mediumNewOpenAITogether AI

Design token-level billing and metering for an LLM API: count at the engine, aggregate, reconcile, and survive the failure modes.

Billing is the one pipeline where a dropped event is money and a duplicated one is a support ticket. Where tokens are counted and by whom, the usage event and its idempotency key, the aggregation that closes a month, the reconciliation that proves the ledger against the engines, and the six ways it goes wrong.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

📐 AI Systems Design

Control Plane and API Design for GPU PlatformsEvery GPU platform has a control plane, and its API is what the rest of the organization experiences as the platform. Three semantics decide whether it survives contact with a network: idempotent creation so a retried request does not launch a second job on sixty-four GPUs, cancellation modelled as intent because only the node agent can stop a running process, and cursor pagination that does not skip rows when work is created during a listing.

Core

💻 Coding for InfraSign in

Retry, Backoff and IdempotencyA retry is a second request that the system did not budget for, and a thousand clients retrying at the same moment is a second outage that the first one caused. The craft is small and specific: retry only what is safe to retry, wait an exponentially growing random interval so the retries spread out, cap the total retries with a budget, and make every retried operation idempotent so a duplicate does not double-charge or double-train. This page derives why synchronized retries double the load, works the jitter arithmetic, implements the client correctly, and covers idempotency keys for the operations an AI platform exposes.

Advanced

💻 Coding for Infra🔒 Premium

Parsing Kernel Traces and LogsThe profiler exported a 40 GB trace; the fleet emitted a terabyte of logs overnight; the interviewer hands you a text file of kernel records and asks which kernels dominated, per GPU, per stream. The problem is a parser plus an aggregation, and it is a test of three habits: streaming instead of loading, choosing the key you aggregate on before you write a line, and handling malformed input as data rather than as an exception. This page works the reported trace-classification problem end to end, derives the memory bounds of each design, and shows the generator-based structure that scales from a screen-sized file to a fleet.

Foundational

📐 AI Systems Design

Inference Platform ArchitectureAn LLM inference platform is the layer between a product's API call and a GPU running a serving engine, and every design round starts from its reference shape: a gateway that authenticates and rate-limits, a router that picks a replica with the right model and a warm cache, a per-replica scheduler that batches, engines that run prefill and decode, a KV cache tier, an autoscaler, and the observability that makes it operable. This page draws that shape, sizes each box for a concrete workload, and walks the derivation from user demand to replica count that every design answer has to contain.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign prefill and decode disaggregation for a 70B serving fleet: pool sizes, KV transfer, routing, and when to turn it on.Next in this trackDesign an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.Next in this trackDesign inference for a 1T-parameter mixture-of-experts model: expert parallelism on an NVL72 rack, the attention cache, balance.

FEDITOR'S NOTE

Scored on counting at the engine with the tokenizer that ran, on an idempotent usage event per request written before the response completes, on aggregation with reconciliation against engine totals, and on handling streams that abort, cached tokens and retries.

DISCUSSION · 0

No comments yet — be the first to share your approach.