← AI Infrastructure System Design / 27

mediumOpenAITogether AI

Design token-level billing and metering for an LLM API: count at the engine, aggregate, reconcile, and survive the failure modes.

Trace usage from execution to one settled charge. Distinguish requests, attempts and events, then handle cancellation, late records, replay and reconciliation.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Trace usage from execution to one settled charge. Distinguish requests, attempts and events, then handle cancellation, late records, replay and reconciliation.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🚀 Inference & Serving🔒 Premium

Tokens, Embeddings and LogitsTrace token IDs, embedding rows and next-token scores. Work out which tensors grow with a batch, when all-position logits waste memory, and why token probabilities do not establish factual confidence.

📐 AI Systems Design

Control Plane and API Design for GPU PlatformsA control plane records intended work and coordinates its execution. Clear retry, cancellation and listing semantics keep its API useful when requests or workers fail.

🩺 Fleet Reliability & Observability🔒 Premium

Tracing Inference RequestsConnect gateway, queue, model execution and streaming delivery without confusing overlapping spans or GPU activity with user-visible latency.

🚀 Inference & Serving🔒 Premium

Voice Turns, Barge-In and Playback BuffersVoice interruption must stop unwanted generation and invalidate queued playback. Track turn identity, bound audio in milliseconds, and distinguish generated, submitted and actually played output.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign prefill and decode disaggregation for a 70B serving fleet: pool sizes, KV transfer, routing, and when to turn it on.Next in this trackDesign an internal LLM gateway for a company: routing across providers and models, caching, fallbacks, and cost controls.Next in this trackDesign inference for a 1T-parameter mixture-of-experts model: expert parallelism on an NVL72 rack, the attention cache, balance.

FEDITOR'S NOTE

Assess authoritative usage evidence, atomic settlement, pricing-policy boundaries and recovery from incomplete or duplicated records.

DISCUSSION · 0

No comments yet — be the first to share your approach.