mediumAnthropicFireworks

How long does prefill take for an 8k-token prompt on a 70B model?

Work through projection and attention FLOPs for an 8,192-token prompt. Separate a peak-based lower bound, an assumed-rate estimate and observed TTFT.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Work through projection and attention FLOPs for an 8,192-token prompt. Separate a peak-based lower bound, an assumed-rate estimate and observed TTFT.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🚀 Inference & Serving

Prefill vs DecodeFollow one prompt through its first prediction and later decode steps, then separate GPU work from the wait a user sees. Learn why batch size and context change the likely bottleneck.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Memory-Bound vs Compute-Bound KernelsUse a worked timing example to distinguish a bandwidth ceiling from a measured bottleneck. Learn which profiler evidence to inspect and why decode, prefill and a whole serving step need separate diagnoses.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationCount the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

Foundational

🧩 GPU & Accelerator Architecture

Roofline ModelDerive a throughput ceiling from work, traffic and hardware limits. Read a measured point below the roof, distinguish memory boundaries, and choose an optimization without promising peak performance.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackEstimate how many H100-hours it took to train Llama 3.1 405B, then check it against the paper Next in this trackSize an inference fleet for a 70B model serving 1,000 concurrent users Next in this trackA MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?

FEDITOR'S NOTE

Trace the example, state its assumptions, and distinguish the estimate from measured behavior.

DISCUSSION · 0

No comments yet — be the first to share your approach.