hardTogether AIBaseten

How big can the decode batch get before the step becomes compute-bound?

Derive the batch-295 shortcut, count the tensors it omits, and test whether the resulting kernel estimate is even inside the replica’s memory budget.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Derive the batch-295 shortcut, count the tensors it omits, and test whether the resulting kernel estimate is even inside the replica’s memory budget.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationCount the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Memory-Bound vs Compute-Bound KernelsUse a worked timing example to distinguish a bandwidth ceiling from a measured bottleneck. Learn which profiler evidence to inspect and why decode, prefill and a whole serving step need separate diagnoses.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

Foundational

🧩 GPU & Accelerator Architecture

Roofline ModelDerive a throughput ceiling from work, traffic and hardware limits. Read a measured point below the roof, distinguish memory boundaries, and choose an optimization without promising peak performance.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA startup wants to train a 7B model on 1 trillion tokens in three weeks. What cluster do they rent?Next in this trackEstimate the activation memory for training a 70B model at an 8k sequence length Next in this trackModel our serving request queue with Little's law. What happens as we approach saturation?

FEDITOR'S NOTE

Assess whether the candidate separates projection intensity, attention work and feasible service concurrency.

DISCUSSION · 0

No comments yet — be the first to share your approach.