hardCoreWeaveTogether AIBaseten

Compare A100, H100 and B200 for a 70B serving fleet. Which gives the most tokens per dollar, and where do fp8 and fp4 change the ranking?

Compare the cost of serving the same workload at the same latency and quality. A memory-traffic estimate explains what each GPU might improve; measured throughput and your actual rental price determine the purchase.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

GPU Generations: A100 to BlackwellGPU generations change memory, arithmetic support and interconnect capacity at different rates. Compare the specification that limits your workload rather than the generation name.

Core

🧩 GPU & Accelerator ArchitectureSign in

Numerics: FP32, BF16, FP8 and FP4Floating-point formats trade range and precision for storage and compute cost. Understand exponent bits, significand bits, scaling and accumulation before choosing a format.

Foundational

🖧 Hardware & Cluster Build-Out

Accelerator Selection: H100 to B300 and RTX PRO 6000Choose an accelerator by checking memory capacity, memory bandwidth and compute throughput separately. Each limits a different part of the workload.

Foundational

🧮 Open Weights & Serving Engines

Weight Formats: FP8 Blocks, MXFP4 and AWQDerive the storage cost of low-precision tensors, including scales, zero points and higher-precision exceptions. Distinguish formats from quantization methods, understand fused dequantization, and test whether a smaller checkpoint actually improves serving capacity without unacceptable quality loss.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackIf you could change one thing about GPU architecture for LLMs, what would it be, and what argues against it?Next in this trackOne node in your cluster shows half the expected NVLink bandwidth in nccl-tests. Walk me through isolating it.Next in this trackHalf the weights are zero. Why did serving not get faster?

FEDITOR'S NOTE

Look for a capacity check, a clearly labelled performance estimate and a cost comparison that counts only output delivered within the latency target. Quantization must pass the same quality evaluation.

DISCUSSION · 0

No comments yet — be the first to share your approach.