Accelerator Selection: H100 to B300 and RTX PRO 6000

TL;DR: Start by checking whether the weights, KV cache and runtime fit in memory. Then identify what limits the workload: small-batch decode often waits on memory, while larger matrix operations can use more of the GPU’s compute. Compare measured performance for your model and batch size before treating a specification ratio as a speedup.

The published numbers, as of September 2026

The table is a dated comparison, not a complete list of purchasable configurations. Check the exact system specification before sizing a deployment. NVIDIA’s HGX table lists sparse FP8 figures for eight GPUs: converting 72 PFLOPS to dense and dividing by eight gives 4.5 PFLOPS per GPU. Per-GPU memory and bandwidth for Blackwell Ultra come from NVIDIA's Blackwell Ultra datasheet, which lists the HGX B300 and GB300 NVL72 GPUs separately, and B200 figures from NVIDIA's HGX reference architecture; checked 2026-10-03. Like B200's 180 GB (announced as 192 GB), these are shipping capacities, not raw HBM size. The H100 specification and RTX PRO 6000 specification use their own product conventions.

NVIDIA lists the RTX PRO 6000 Server Edition at 2 PFLOPS of FP8 Tensor Core throughput without saying whether that figure is dense or sparse, so it is left out of the dense column rather than compared as if it were dense.

Two things in that table decide more deployments than the headline FLOPS. In the HGX configurations, the B300 carries 270 GB where the B200 carries 180 GB, at the same dense FP8 peak: Blackwell Ultra's extra memory is for larger models and longer KV caches. It does add arithmetic in other formats; dense FP4 rises by about half, so "same FLOPS" is true only for the FP8 comparison. And the RTX PRO 6000 has no NVLink at all, which is not a detail. It means multi-GPU work on that part crosses PCIe Gen5, and that changes the answer to every parallelism question.

The three gates, each with its arithmetic

Consider a hypothetical dense 70B model. BF16 weights use two bytes per parameter, before runtime allocations:

weight payload = 70e9 parameters x 2 bytes = 140 GB
headroom after weights:
  80 GB device:  does not fit
  180 GB device: 40 GB before runtime and request state
  270 GB device: 130 GB before runtime and request state  (HGX B300; GB300's 279 GB leaves 139 GB)
sanity: aggregate free memory is not enough; each supported shard must fit its device.

A capacity check comes before a speed estimate. For a model that fits, a weight-only, batch-one decode bound divides memory bandwidth by weight bytes. At an assumed 7.7 TB/s (the HGX B300 figure), that gives 7.7e12 / 140e9 = 55 tokens/s. Attention state, other traffic, compute and overhead can lower the result. It is not a measured serving rate.

Prefill exposes more matrix work. A rough dense-projection count for a 4,096-token prompt is 2 x 70e9 x 4,096 = 5.73e14 FLOPs. With an illustrative achieved rate of 40% of a 4.5 PFLOPS FP8 dense peak, that portion takes about 0.319 s. This assumes a supported FP8 prefill path that meets the quality requirements; at 40% of the 2.25 PFLOPS dense BF16 peak, the same work takes about 0.637 s. Attention and scheduling add work beyond either estimate.

Compare a complete request by adding the times of its phases. A ratio between specification peaks does not establish the speedup of the whole request.

Why the ridge point keeps moving

The ridge point is the compute peak divided by the bandwidth of the memory boundary being modeled. Use a consistent precision and dense or sparse convention:

H100 FP8 dense: 1.979e15 FLOP/s / 3.35e12 bytes/s = 591 FLOP/byte
B200 FP8 dense: 4.5e15 FLOP/s / 8e12 bytes/s = 563 FLOP/byte
sanity: the sparse B200 peak would double the numerator and describe a different roof.

The similar values (within about 5%) are a useful warning against assuming that every generation shifts the boundary in one direction. Read Memory-Bound vs Compute-Bound Kernels for how to turn a roofline estimate into a profiling hypothesis. Kernel shape, effective bandwidth and supported numerical formats still matter.

Where the RTX PRO 6000 fits, and where it does not

A model that fits on one RTX PRO 6000 can be served without cross-card communication. For example, a hypothetical 30B FP8 payload is about 30 GB before scales and higher-precision tensors. A 96 GB device then has room to consider runtime and request state, but the remaining 66 GB is not automatically all KV cache.

For a larger model, PCIe-connected cards remain an option. Compare the actual collective latency and throughput rather than multiplying an NVLink measurement by a bandwidth ratio. Frequent small collectives can be dominated by synchronization, and devices may share upstream paths. Independent replicas can be preferable when they satisfy the model and service requirements.

What interviewers are listening for

Explain which resource limits your workload before choosing a GPU. For a decode-heavy service, memory capacity and bandwidth may matter more than peak tensor throughput. For training, include communication and achieved compute throughput. State the model, precision and batch size behind the comparison, and date the hardware specifications.

Key takeaways

• Check capacity first, then use the workload’s arithmetic intensity to decide whether bandwidth or compute is the likely limit.
• As checked on 2026-10-03, NVIDIA publishes 80 GB at 3.35 TB/s for H100 SXM, 180 GB at up to 8 TB/s for B200 in HGX, 270 GB at 7.7 TB/s for HGX B300 (279 GB at 8 TB/s in GB300 NVL72), and 96 GB at 1.597 TB/s for RTX PRO 6000 Server Edition.
• Bandwidth divided by weight bytes gives a simplified batch-one decode bound. A 70B BF16 model needs about 140 GB for weights alone, so it does not fit on one 80 GB H100.
• With the dense peaks used here, the FP8 ridge points are about 591 FLOP/byte for H100 and 563 FLOP/byte for B200. Mixing dense and sparse peaks would distort the comparison.
• The RTX PRO 6000 has no NVLink. Models can still be split across cards, but PCIe communication may make independent replicas a better choice.