← Napkin Math, Cost & Capacity / 24

hardAnthropicFireworks

Estimate the latency of one decode step for a 70B model under tensor parallelism across eight H100s

Estimate a TP8 decode step from weights, cache reads and explicitly assumed collective times, then replace the serial worksheet with a measured critical path.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Estimate a TP8 decode step from weights, cache reads and explicitly assumed collective times, then replace the serial worksheet with a measured critical path.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits operations within a layer across GPUs. Derive the column/row matrix split, count training and inference collectives, and choose a group size from memory, latency and actual interconnect performance.

🔌 Networking & Storage

Debugging a Slow All-ReduceInterpret NCCL algbw and busbw, then isolate a slow collective with controlled tests. Distinguish rank delays, GPU-memory paths and shared-fabric limits without guessing from bandwidth ratios.

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

🕸️ Distributed Training

Collective Communication PrimitivesFollow small vectors through all-reduce, reduce-scatter, all-gather and all-to-all. Separate the result each worker needs from the algorithm-specific byte count and measured completion time.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow big can the decode batch get before the step becomes compute-bound?Next in this trackA startup wants to train a 7B model on 1 trillion tokens in three weeks. What cluster do they rent?Next in this trackEstimate the activation memory for training a 70B model at an 8k sequence length

FEDITOR'S NOTE

Separate bandwidth floors from benchmarks, test feasibility and avoid summing overlapping trace intervals.

DISCUSSION · 0

No comments yet — be the first to share your approach.