Accelerator Selection: H100 to B300 and RTX PRO 6000

TL;DR: Read three numbers per part and treat them separately. Capacity decides whether the model and its KV cache fit at all, and no amount of compute rescues a part that cannot hold the weights. Bandwidth decides decode throughput, because generating one token reads every active weight once, so tokens per second per GPU is bounded by bandwidth divided by bytes read. Tensor FLOPS decide prefill and training, which are compute-bound at any reasonable batch size. Those three have not scaled together. From H100 to B300, NVIDIA's published capacity per GPU grew 3.6 times and bandwidth 2.4 times while FP8 tensor throughput grew about 4.5 times, so the ridge point, meaning the arithmetic intensity where a kernel stops being bandwidth-bound, moved from 591 FLOP per byte to over 1,100. Newer parts are relatively hungrier for arithmetic, which changes batch sizes, quantization choices and what "fully utilized" means.

The published numbers, as of September 2026

Every figure in this table comes from NVIDIA's own product pages and datasheets. The per-GPU figures for the HGX platforms are the node totals divided by the eight GPUs in a node, and that is stated so a reader can check it.

Two things in that table decide more deployments than the headline FLOPS. The B300 carries 288 GB where the B200 carries 180 GB, at the same FP8 throughput, which tells you what Blackwell Ultra was built for: holding larger models and longer KV caches rather than doing more arithmetic per second. And the RTX PRO 6000 has no NVLink at all, which is not a detail. It means multi-GPU work on that part crosses PCIe Gen5, and that changes the answer to every parallelism question.

The three gates, each with its arithmetic

gate 1: capacity, which decides whether it runs at all
  a 70B model in bf16 = 70e9 x 2 B = 140 GB of weights
  on H100 (80 GB):   does not fit on one; needs 2 at minimum, 4 in practice with KV cache
  on B200 (180 GB):  fits with 40 GB left for KV cache and activations
  on B300 (288 GB):  fits with 148 GB left, which is roughly 460k tokens of KV at the
                     corpus figure of 320 KB per token for a 70B GQA model
  on RTX PRO 6000 (96 GB): does not fit in bf16; fits in FP8 at 70 GB with 26 GB spare

gate 2: bandwidth, which decides decode
  decode at small batch reads every active weight once per token, so
    tokens/s per GPU <= bandwidth / bytes of active weights
  70B in bf16, 140 GB read per token:
    H100:            3.35e12 / 140e9  = 24 tokens/s
    B200:            7.7e12  / 140e9  = 55 tokens/s
    B300:            8e12    / 140e9  = 57 tokens/s
    RTX PRO 6000:    1.597e12 / 140e9 = 11 tokens/s, and it does not fit anyway
sanity: B200 is roughly 4.5 times the H100's FP8 arithmetic and 2.3 times its bandwidth, and
        decode tracks the 2.3, which is why a decode-heavy fleet does not see the headline
        speedup

gate 3: tensor FLOPS, which decides prefill and training
  prefill of a 4,096-token prompt on a 70B model
    FLOPs = 2 x params x tokens = 2 x 70e9 x 4,096 = 5.7e14
    at 40% of peak FP8:
      H100:  5.7e14 / (1.979e15 x 0.4) = 0.72 s
      B200:  5.7e14 / (9e15 x 0.4)     = 0.16 s
  so the same fleet change that gives 2.3x on decode gives about 4.5x on prefill
sanity: a workload's speedup from a hardware upgrade is a weighted average of those two, and
        the weights come from your own prefill-to-decode token ratio, not from a vendor slide

Why the ridge point keeps moving

ridge point = peak FLOPS / peak bandwidth, in FLOP per byte
  H100 at bf16:   989e12  / 3.35e12 = 295 FLOP/B     (the corpus anchor)
  H100 at FP8:    1.979e15 / 3.35e12 = 591 FLOP/B
  B200 at FP8:    9e15    / 7.7e12  = 1,169 FLOP/B
  B300 at FP8:    9e15    / 8e12    = 1,125 FLOP/B
  RTX PRO 6000 at FP8: 2e15 / 1.597e12 = 1,252 FLOP/B
sanity: every generation raises the bar a kernel must clear to be compute-bound, so more
        kernels are memory-bound on newer hardware than on older, which is the opposite of
        what most people assume when they upgrade

The practical consequence is that the operations worth fusing, the batch sizes worth running and the value of quantization all shift with the part. Memory-Bound vs Compute-Bound Kernels covers the classification; the point here is that the boundary is a property of the hardware you bought, and it moved.

Where the RTX PRO 6000 fits, and where it does not

This part is the one people misjudge in both directions. NVIDIA publishes 96 GB of GDDR7 at 1,597 GB/s, 2 PFLOPS of FP8, up to 600 W, PCIe Gen5, and MIG partitioning into up to four instances. What it does not have is NVLink.

what that means in practice
  one model per card, replicated, is the shape this part is good at
    a 30B model in FP8 (30 GB) leaves 66 GB for KV cache on one card, no parallelism needed
    four MIG instances per card serve four small models or four tenants with isolation
  tensor parallelism across two cards crosses PCIe Gen5 x16, about 64 GB/s each direction
    against 900 GB/s of NVLink on an H100 and 1.8 TB/s on a Blackwell SXM part
    a tensor-parallel all-reduce that costs 1 ms on NVLink costs roughly 14 to 28 times more
    here, which for decode, where an all-reduce happens twice per layer per token, is fatal
  the arithmetic that decides it
    per-token TP all-reduce volume for a 70B model, TP=2: about 2 x hidden x 2 B per layer
    at hidden 8,192 and 80 layers: 80 x 2 x 8,192 x 2 B = 2.6 MB per token
    on PCIe at 64 GB/s that is 41 microseconds of pure communication per token, against a
      total decode budget of maybe 20 ms, so it is survivable at TP=2 and not at TP=8
sanity: the part is a good fit for many replicas of a model that fits on one card, and a poor
        fit for one large model split across cards, and that sentence is the whole selection
        rule for it

What interviewers are listening for

They want to hear the three gates kept separate. A candidate who answers "B200 is faster" has said nothing; a candidate who says "for our decode-heavy mix the gain is bandwidth-limited at about 2.3 times, and the FP8 FLOPS gain of 4.5 times only shows up in prefill" has done the work. The second thing they listen for is whether you know what the part cannot do: no NVLink on the RTX PRO 6000, and the 288 GB on B300 being a capacity story rather than a compute one. The third is dating: hardware facts decay, so say when your numbers are from and where.

Key takeaways

• Three independent gates: capacity decides what fits, bandwidth decides decode, tensor FLOPS decide prefill and training.
• As of September 2026, NVIDIA publishes 80 GB at 3.35 TB/s for H100 SXM, 180 GB at 7.7 TB/s for B200, 288 GB at 8 TB/s for B300, and 96 GB at 1.597 TB/s for RTX PRO 6000 Server Edition.
• Decode tokens per second per GPU is bounded by bandwidth divided by active weight bytes: 24 on H100 and 55 on B200 for a 70B model in bf16.
• The FP8 ridge point moved from 591 FLOP per byte on H100 to about 1,169 on B200, so more kernels are memory-bound on newer parts.
• The RTX PRO 6000 has no NVLink, so it suits many replicas of a model that fits on one card and not one model split across cards.