Your fleet is decode-heavy. Is a B300 worth 1.4 times a B200's power for 1.6 times the memory?

TL;DR: Compare on the two numbers decode actually uses. Per NVIDIA's published figures the B200 in an HGX node carries 180 GB at 7.7 TB/s and the B300 carries 288 GB at 8 TB/s, with the same 9 PFLOPS of dense FP8 per GPU. So the B300 offers 60 percent more memory and 4 percent more bandwidth. Decode throughput at a fixed batch size is bandwidth-bound, so the direct gain is that 4 percent and nothing more. The real gain is indirect: 60 percent more memory holds a larger KV cache, which supports a larger batch, and larger batches amortize the weight read across more tokens. For a memory-constrained deployment that is a large win, and for one already running at its target batch size it is close to nothing. Decide by asking whether the current deployment is limited by KV capacity or by bandwidth, and the way to find out is to look at whether the cache is full while GPUs still have headroom.

How to approach it

Establish which resource is binding today before comparing parts, because the answer changes completely. Then compute the direct bandwidth gain, which is small. Then compute the indirect capacity gain through batch size, which is where the case lives. Then compare cost per million tokens rather than cost per GPU. Close with the case that flips it.

A strong answer

A typical situation: a serving team runs a 70B model on 8 H100s per replica and their dashboard shows the KV cache pool at 97 percent utilization with GPU compute at 40 percent. They are being asked whether to buy B200 or B300 for the refresh, and the vendor material talks entirely about FP8 throughput.

The direct comparison, on the numbers decode uses:

direct decode gain, at a fixed batch size
  decode reads the active weights once per step regardless of batch, so throughput scales
  with bandwidth
    H100 -> B200:  7.7 / 3.35 = 2.30x
    B200 -> B300:  8.0 / 7.7   = 1.04x
  the FP8 number is identical between B200 and B300, so it contributes nothing here

indirect gain, through capacity
  KV pool per GPU ≈ (memory x utilization) - weights per GPU - workspace
  a 70B model in FP8, 70 GB, TP=8, so 8.75 GB of weights per GPU
    B200:  180 x 0.92 - 8.75 - 10 = 147 GB per GPU, 1,176 GB across 8
    B300:  288 x 0.92 - 8.75 - 10 = 246 GB per GPU, 1,968 GB across 8
  at the corpus figure of 320 KB per token for a 70B grouped-query model:
    B200:  1,176e9 / 327,680 = 3.59M tokens of cache
    B300:  1,968e9 / 327,680 = 6.01M tokens
  at 8,192 tokens per sequence:
    B200:  438 concurrent sequences
    B300:  734 concurrent sequences, a 1.67x increase
sanity: the capacity gain (1.67x in concurrency) is far larger than the bandwidth gain
        (1.04x), so the B300's case is entirely about how many sequences share each weight
        read

How capacity turns into throughput:

decode throughput with batching
  per step, the engine reads the active weights once and produces one token per sequence
  so tokens per second ≈ (bandwidth / active weight bytes) x batch size, until compute or
    another limit binds
  at 70B in FP8, 70 GB read per step:
    B200 at batch 438:  (7.7e12 / 70e9) x 438 = 110 x 438 = 48,180 tok/s
    B300 at batch 734:  (8.0e12 / 70e9) x 734 = 114 x 734 = 83,676 tok/s
    ratio = 1.74x
  and the power ratio is 1,400 / 1,000 = 1.40x
sanity: 1.74x throughput for 1.40x power is a real efficiency gain, and it exists only
        because the deployment was capacity-limited. Run the same comparison on a workload
        with 512-token contexts and the B200's cache is already big enough, the batch size is
        set by something else, and the gain collapses to the 1.04x bandwidth difference

Accelerator Selection: H100 to B300 and RTX PRO 6000 covers the three gates in general. Bandwidth-Bound Decode Throughput covers why the weight read is the term that matters and how batching amortizes it.

The measurement that decides it, before buying anything:

what to look at on the current fleet
  KV cache utilization             vLLM logs the pool usage; SGLang reports token usage
                                   near 1.0 with requests queued means capacity-bound
  GPU compute utilization          DCGM_FI_PROF_SM_ACTIVE well below 1 alongside a full
                                   cache confirms it
  achieved batch size              compare against what the SLO would allow
  p99 TTFT under load              rising because requests queue for cache, not for compute
sanity: a full cache with idle SMs is the signature that says buy memory; a full cache with
        busy SMs says the workload is compute-bound somewhere and more memory will not help

The reversal condition: if the deployment is not capacity-limited, the B200 is the better purchase and it is not close. Same FP8 throughput, 96 percent of the bandwidth, and 71 percent of the power for a part that costs less. Short-context serving, workloads with heavy prefix caching that keeps effective KV small, and models with compressed-latent attention whose KV per token is a fraction of the classic figure all fall here. Multi-Head Latent Attention and Sparse Indexers covers the last case. Newer open-weights models deliberately reduce KV per token, which weakens the capacity argument for the whole class of part over time.

What interviewers probe next

• "Why doesn't the FP8 number matter?" Decode at any realistic batch is bandwidth-bound; the arithmetic is a small fraction of the step. It matters for prefill, which is a different pool's problem.
• "What limits batch size other than KV?" The latency target, since a larger batch raises per-token latency for everyone in it, and at some point the SLO binds before memory does.
• "Would you mix parts?" Yes, and that is often the answer: a prefill pool of compute-rich parts and a decode pool of memory-rich ones, which is what disaggregation enables.
• "How does power factor in?" 1,400 W against 1,000 W changes rack density and cooling, so the comparison is per rack rather than per GPU once the facility is fixed.

Common mistakes

• Comparing on FP8 FLOPS, which are identical between these two parts and irrelevant to decode anyway.
• Treating the memory gain as a direct throughput gain rather than as a batch-size gain.
• Ignoring the 1.4 times power ratio, which changes GPUs per rack and therefore the facility footprint.
• Not checking whether the current deployment is capacity-limited before buying capacity.
• Applying the analysis to a model whose attention design already makes KV small.

Key takeaways

• B200 and B300 have identical published FP8 throughput; the B300 offers 60 percent more memory and 4 percent more bandwidth.
• Decode at fixed batch scales with bandwidth, so the direct gain is 1.04 times.
• Capacity converts to throughput through batch size: 438 to 734 concurrent sequences on a 70B model, giving about 1.74 times throughput for 1.40 times power.
• Confirm the deployment is capacity-bound first: a full KV pool with idle SMs is the signature.
• Compressed-latent attention shrinks KV per token by tens of times, which weakens the memory argument for future models.