The common eight-GPU training-node H100. Its fast local GPU fabric is separate from whatever network connects the node to another node.
Memory per accelerator
80 GB
HBM3
Memory bandwidth
3.35 TB/s
Published peak, not measured application throughput
Remember this
GCP A3 Mega, High and Edge all use H100 SXM. Their network bandwidth and placement differ. “Mega” is not a new Hopper chip, and “Edge” does not mean a different 80 GB memory system.
When this is a sensible choice
Start here if…
Start here for a mature CUDA training or serving stack needing tensor parallelism. Eight 80 GB devices provide 640 GB nominal aggregate memory, but every shard still has to fit its assigned device.
Choose another configuration if…
H200 is worth comparing when H100 is memory-limited; a smaller GPU can be better for a small independent service. Profile queueing and attention work before concluding that an H100 needs replacement.
Specifications with their boundaries attached
Architecture
Hopper
Memory
80 GB HBM3
Memory bandwidth
3.35 TB/s per accelerator
Peak compute
989.5 TFLOPS BF16/FP16; 1,979 TFLOPS FP8, dense
Scale-up interconnect
NVLink 4: 900 GB/s bidirectional per GPU; HGX uses NVSwitch
Host attachment
PCIe 5.0 x16
Power
Up to 700 W
Partitioning
Up to seven MIG instances
Catalogue status
Documented product
Compute figures are theoretical peaks at the stated precision. Dense and structured-sparse rates must not be mixed. Bandwidth labelled bidirectional combines both directions. See the source documents.
Follow the bytes · conceptual topology
Three bandwidths, three different jobs
Local memory80 GB HBM3
↔
Compute enginesExecute kernels on these bytes
① Memory bandwidth: 3.35 TB/s
Accelerator AOwn local memory
↔
Accelerator BOwn local memory
② Scale-up: NVLink, Infinity Fabric or PCIe NVLink 4: 900 GB/s bidirectional per GPU; HGX uses NVSwitch
Server AAccelerators + host
↔
Server BAnother fabric domain
③ Scale-out: NICs + switches + placement InfiniBand, RoCE, EFA or provider-specific transport
A 400 Gb/s NIC has a 50 GB/s raw line-rate equivalent before overhead. A 900 GB/s bidirectional NVLink figure counts traffic in both directions. Neither is the bandwidth at which a GPU reads its own HBM. This diagram explains the boundaries; it is not a wiring diagram for a particular cloud machine.
Where it appears in provider documentation
Documented configurations, checked September 12, 2026. Listing does not guarantee regional stock, quota, allocation size or an on-demand purchase.
320B · total parameter proxy · 18B active in the main model
320B is the model card’s total; 18B active is not its storage size. The vLLM recipe reports about 306 GiB for the native FP8 checkpoint. Hopper requires BF16 KV for this model; the documented ROCm path targets gfx950, not every Instinct GPU.
Weights verified; precise release day not certified in this reference.
What is the memory floor?
Start with total parameters, then add the memory the workload needs. This arithmetic does not certify a serving configuration. All output sizes below are decimal GB.
Override device capacity with the memory exposed by your allocation, particularly for cloud B300 and partitioned devices. The starting 32 GB budget and 15% reserve are editable teaching assumptions. They are not measurements for the selected model. Mixed-precision tensors, quantization scales, vision encoders and draft models can increase the weight payload.
320 GB
Raw weights only 320B × 8 bits ÷ 8
68 GB
Budget per device 80 GB × (1 − 15%)
6 devices
Arithmetic lower bound round up ((weights + 32) ÷ budget)
WeightsCache + runtimeRemaining budget
This assumes perfectly balanced sharding. A result of three does not prove that a three-device parallel layout is supported. Check layer/expert divisibility, actual allocatable memory, precision kernels and the fabric before renting.
320B is the model card’s total; 18B active is not its storage size. The vLLM recipe reports about 306 GiB for the native FP8 checkpoint. Hopper requires BF16 KV for this model; the documented ROCm path targets gfx950, not every Instinct GPU. Read the model source ↗
For full training, also budget gradients, optimizer states, activations and communication buffers. The inference weight estimate above is insufficient.
Tokens per second, TTFT and TPOT are not certified for this hardware in our reference. Use the benchmark checklist to compare an exact model, software revision and workload.
Nearby memory capacities, different tradeoffs
These are comparison candidates selected by memory capacity, not performance rankings or drop-in replacements.
80 GB of HBM on the SXM4 form factor. A100 remains useful for BF16 training and supported inference, but cannot execute Hopper’s native FP8 Tensor Core path.
80 GB of HBM on the PCIe form factor. A100 remains useful for BF16 training and supported inference, but cannot execute Hopper’s native FP8 Tensor Core path.
The 80 GB PCIe H100 is a different operating point from the SXM card. The same H100 name does not promise 3.35 TB/s memory bandwidth.
Sources and review scope
Manufacturer and cloud documentation checked 2026-09-12. We reviewed specifications and the stated product boundaries; we did not run training or serving benchmarks on this device. Cloud memory and availability can differ by configuration.
Include your access date when citing a changing specification. Link to the specification section for hardware figures or the explanation for a sizing or workload decision.
Research method and limits
We compile manufacturer specifications, cloud documentation, model cards and serving recipes. The reference preserves source units and distinguishes individual devices from nodes and racks. Conflicting figures and unknown fields remain labelled.
Our contribution is the comparison, unit reconciliation, worked arithmetic and workload explanation. Published peaks are vendor specifications. Calculator results are estimates under the displayed assumptions. Neither is a measurement from our own accelerator lab.
For a manufacturer’s specification, consult the original source documents. Cite our page when using its analysis, and retain primary-source attribution for underlying figures. This curated reference does not establish market share, live availability or a universal performance ranking.
Found a discrepancy? Send a correction with the page URL, exact variant, disputed figure and supporting primary document.
