H100 SXM, PCIe and NVL
SXM is a module used in tightly connected server platforms. H100 PCIe is a different 80 GB configuration. H100 NVL has 94 GB per GPU; a bridged pair has 188 GB combined. Memory bandwidth, power and connectivity differ.
Hardware field guide · free reference · reviewed September 12, 2026
GPU & TPU hardware comparison
An H100 is a chip. “A3 Mega” describes a cloud machine around it. Learn which specifications change your model’s memory budget, which links carry its tensors, and what to check before choosing a GPU or TPU.
Follow the bytes · conceptual topology
Three bandwidths, three different jobs
Local memoryWeights + KV cache + activations
Compute enginesExecute kernels on these bytes
① Memory bandwidth: inside each accelerator
Accelerator AOwn local memory
Accelerator BOwn local memory
② Scale-up: NVLink, Infinity Fabric or PCIe
Moves sharded tensors and collective results between devices
Moves sharded tensors and collective results between devices
Server AAccelerators + host
Server BAnother fabric domain
③ Scale-out: NICs + switches + placement
InfiniBand, RoCE, EFA or provider-specific transport
InfiniBand, RoCE, EFA or provider-specific transport
Start with memory
Will it load with room to work?
A 70B model at two bytes per parameter needs about 140 GB for weights alone. Two 80 GB GPUs provide 160 GB nominal capacity, leaving 20 GB before runtime, quantization metadata and request state. That is a budget to investigate, not a deployment guarantee.
Work through your model →Then follow the traffic
Where will the bytes move?
Local memory bandwidth helps feed each accelerator. Links between accelerators move sharded tensors. The cluster network connects servers. A strong number on one of those paths cannot compensate for a bottleneck on another.
Compare the cloud configurations →Browse 37 accelerator profiles. Specs come from linked manufacturer and provider documents. Workload choices are engineering guidance; timing results require a benchmark.
New to the hardware vocabulary? Start with these labels.
- HBM / GDDR
- Memory attached to the accelerator. Capacity tells you how much fits; bandwidth tells you how quickly bytes can move. HBM means high-bandwidth memory; GDDR is graphics double data rate memory.
- BF16 / FP8 / FP4
- Number formats using roughly 16, 8 or 4 bits per value. Fewer bits can reduce storage and work, but the exact format, scales, kernels and model accuracy still matter.
- KV cache
- The stored keys and values used by attention to reuse earlier token computations. More active requests or longer contexts often need more cache.
- NVLink / NVSwitch
- NVIDIA’s links between accelerators and the switches used to connect larger groups. A bridge between two cards and a switched server are different topologies.
- ICI / DCN
- Google’s inter-chip interconnect connects TPU chips within a slice. The data-center network connects separate slices.
- InfiniBand / RoCE / EFA
- Cluster communication technologies. RoCE carries remote direct memory access over Ethernet; EFA is AWS’s Elastic Fabric Adapter. A NIC is the network interface that joins a server to its network.
- Dense / sparse
- Dense compute figures do not assume skipped weights. Structured-sparse peaks depend on a supported sparsity pattern. Keep those two kinds of peak separate.
- TTFT / TPOT
- Time to first token measures the initial wait. Time per output token describes subsequent generation, with an explicit averaging definition.
01 · Hardware specifications
Compare the exact variant
Open a profile for precision-labelled compute, power, partitioning, interconnect and workload tradeoffs. Consumer cards are included for orientation; server hardware is the focus.
37 of 37 accelerators · select up to four to compare
| Compare | Accelerator | Memory | Memory bandwidth | Useful starting point |
|---|---|---|---|---|
| NVIDIA T4 Turing | 16 GB GDDR6 | 0.32 TB/s | A small inference GPU for a model that already fits. Its 16 GB capacity is usually the first limit a new LLM user encounters. Documented product | |
| NVIDIA A10 Ampere | 24 GB GDDR6 | 0.6 TB/s | A 24 GB graphics-capable server GPU. It offers more memory than T4, without the HBM or switched links of A100. Documented product | |
| NVIDIA L4 Ada Lovelace | 24 GB GDDR6 | 0.3 TB/s | A low-power 24 GB inference and media GPU. More compute does not imply more memory bandwidth: its published 300 GB/s is below A10. Documented product | |
| NVIDIA L40S Ada Lovelace | 48 GB GDDR6 | 0.86 TB/s | A 48 GB server card for inference, fine-tuning and graphics. The extra capacity is useful; it does not bring an HGX interconnect. Documented product | |
| NVIDIA A100 40 GB SXM Ampere | 40 GB HBM2 | 1.56 TB/s | 40 GB of HBM on the SXM4 form factor. A100 remains useful for BF16 training and supported inference, but cannot execute Hopper’s native FP8 Tensor Core path. Documented product | |
| NVIDIA A100 80 GB SXM Ampere | 80 GB HBM2e | 2.04 TB/s | 80 GB of HBM on the SXM4 form factor. A100 remains useful for BF16 training and supported inference, but cannot execute Hopper’s native FP8 Tensor Core path. Documented product | |
| NVIDIA A100 80 GB PCIe Ampere | 80 GB HBM2e | 1.94 TB/s | 80 GB of HBM on the PCIe form factor. A100 remains useful for BF16 training and supported inference, but cannot execute Hopper’s native FP8 Tensor Core path. Documented product | |
| NVIDIA H100 80 GB SXM Hopper | 80 GB HBM3 | 3.35 TB/s | The common eight-GPU training-node H100. Its fast local GPU fabric is separate from whatever network connects the node to another node. Documented product | |
| NVIDIA H100 80 GB PCIe Hopper | 80 GB HBM2e | 2 TB/s | The 80 GB PCIe H100 is a different operating point from the SXM card. The same H100 name does not promise 3.35 TB/s memory bandwidth. Documented product | |
| NVIDIA H100 NVL Hopper | 94 GB HBM3 | 3.9 TB/s | H100 NVL is a 94 GB PCIe GPU often discussed as a two-card, 188 GB pair. Always count how many cards the listing includes. Documented product | |
| NVIDIA H200 SXM Hopper | 141 GB HBM3e | 4.8 TB/s | Hopper with a larger, faster memory system: 141 GB at 4.8 TB/s. Its value over H100 is easiest to see in memory capacity and memory traffic. Documented product | |
| NVIDIA B200 HGX Blackwell | 180 GB HBM3e | 8 TB/s | An HBM-rich Blackwell GPU for large training and inference jobs. Use the 180 GB HGX usable-memory specification, not an early 192 GB announcement. Documented product | |
| NVIDIA B300 HGX Blackwell Ultra | 288 GB HBM3e | 8 TB/s | Blackwell Ultra’s larger HBM budget helps large models and long context. A cloud allocation may expose less memory than the 288 GB physical-product figure. Documented product | |
| NVIDIA GB200 NVL72 Grace Blackwell | 186 GB HBM3e | 8 TB/s | A rack-scale system, not a single PCIe card. This comparison uses 186 GB per GPU from GCP’s four-GPU VM configuration; the full rack includes 72 GPUs and 36 Grace CPUs. Documented product | |
| NVIDIA GB300 NVL72 Grace Blackwell Ultra | 279 GB HBM3e | 8 TB/s | A rack-scale system, not a single PCIe card. This comparison uses 279 GB per GPU from GCP’s four-GPU VM configuration; the full rack includes 72 GPUs and 36 Grace CPUs. Documented product | |
| RTX PRO 6000 Blackwell Server Edition Blackwell | 96 GB GDDR7 | 1.6 TB/s | 96 GB of GDDR7 makes this an interesting inference and mixed AI/graphics card. Its memory capacity sits above H100, but its bandwidth does not. Documented product | |
| AMD Instinct MI300X CDNA 3 | 192 GB HBM3 | 5.3 TB/s | 192 GB of HBM on an AMD CDNA 3 accelerator. The memory is useful only after the model’s kernels and serving engine work on the ROCm target. Documented product | |
| AMD Instinct MI325X CDNA 3 | 256 GB HBM3e | 6 TB/s | 256 GB of HBM on an AMD CDNA 3 accelerator. The memory is useful only after the model’s kernels and serving engine work on the ROCm target. Documented product | |
| AMD Instinct MI355X CDNA 4 | 288 GB HBM3e | 8 TB/s | 288 GB of HBM on an AMD CDNA 4 accelerator. The memory is useful only after the model’s kernels and serving engine work on the ROCm target. Documented product | |
| Google TPU v4 TPU v4 | 32 GiB HBM | 1.2 TB/s | A mature training-oriented TPU with 32 GiB per chip. Existing XLA jobs can be a better starting point than porting a CUDA-only workload. Documented product | |
| Google TPU v5e TPU v5e | 16 GB HBM | 0.86 TB/s | The smaller-memory TPU branch for economical training and serving. Eight chips provide eight 16 GB budgets, not one shared 128 GB allocation. Documented product | |
| Google TPU v5p TPU v5p | 95 GiB HBM | 2.77 TB/s | A large-pod training TPU with 95 GiB per chip, much more memory than v5e. The suffix identifies a different design, not a larger v5e VM. Documented product | |
| Google TPU v6e (Trillium) TPU v6e (Trillium) | 32 GB HBM | 1.64 TB/s | Trillium doubles v5e’s nominal memory to 32 GB and increases BF16 compute. It retains a 256-chip, 2D pod design. Documented product | |
| Google TPU7x (Ironwood) TPU7x (Ironwood) | 192 GiB HBM | 7.38 TB/s | Ironwood adds a much larger HBM budget and native FP8. Google’s technical name is TPU7x; its two chiplets are visible as two framework devices per chip. Documented product | |
| Google TPU 8t TPU 8t | 216 GB HBM | 6.53 TB/s | Google’s announced eighth-generation training design. It emphasizes large-scale pre-training, a 3D torus and a new scale-out network. Announced; orderability not verified | |
| Google TPU 8i TPU 8i | 288 GB HBM | 8.6 TB/s | Google’s announced eighth-generation serving design. Larger SRAM and Boardfly target communication and state movement in reasoning workloads. Announced; orderability not verified | |
| AWS Trainium2 NeuronCore-v3 | 96 GiB HBM | 2.9 TB/s | AWS’s own training accelerator, also used for supported inference. It runs the Neuron stack rather than a CUDA binary. Documented product | |
| GeForce RTX 5090 Blackwell | 32 GB GDDR7 | 1.79 TB/s | A 32 GB desktop card for local learning and supported small-model inference. The memory limit remains real even when the compute headline looks large. Documented product | |
| GeForce RTX 5080 Blackwell | 16 GB GDDR7 | 0.96 TB/s | A 16 GB desktop card for local learning and supported small-model inference. The memory limit remains real even when the compute headline looks large. Documented product | |
| GeForce RTX 5070 Blackwell | 12 GB GDDR7 | 0.67 TB/s | A 12 GB desktop card for local learning and supported small-model inference. The memory limit remains real even when the compute headline looks large. Documented product | |
| NVIDIA BlueField-3 DPU Infrastructure offload | Not a model-memory device | N/A | A DPU moves infrastructure work such as networking, security and storage off the host. It is not another name for Google TPU. Documented product | |
| AWS Inferentia2 NeuronCore-v2 | 32 GiB HBM | 0.88 TB/s | An inference-focused AWS chip with two NeuronCore-v2 cores. Inf2 instances can contain up to twelve chips. Documented product | |
| AWS Trainium3 NeuronCore-v4 | 144 GiB HBM | 4.9 TB/s | Eight NeuronCore-v4 cores and 144 GiB of HBM per chip. Its low-precision and memory upgrades are much larger than its dense BF16 compute increase over Trainium2. Documented product | |
| NVIDIA A30 Ampere | 24 GB HBM2 | 0.93 TB/s | A 24 GB HBM2 accelerator for shared enterprise servers. It has more memory bandwidth than L4, but an older compute architecture. Documented product | |
| NVIDIA H200 NVL Hopper | 141 GB HBM3e | 4.8 TB/s | H200 memory capacity in a PCIe card for air-cooled servers. It retains 141 GB and 4.8 TB/s, with a lower compute peak than H200 SXM. Documented product | |
| RTX PRO 6000 Blackwell Workstation Edition Blackwell | 96 GB GDDR7 ECC | 1.79 TB/s | A desktop card with 96 GB of memory for local model work and graphics. Its 1.792 TB/s specification differs from the Server Edition’s 1.597 TB/s. Documented product | |
| AMD Instinct MI350X CDNA 4 | 288 GB HBM3e | 8 TB/s | MI350X and MI355X share 288 GB of HBM3e and 8 TB/s peak bandwidth. MI350X has a lower board-power specification and lower compute peaks. Documented product |
The names that cause expensive mix-ups
Mega, High and Edge are cloud machines
Google’s eight-GPU A3 High, Mega and Edge machines all use H100 SXM 80 GB. Their documented maximum VM network rates differ: 1,000, 1,800 and 400–600 Gb/s. Those labels do not create new H100 silicon.
Google’s machine specifications ↗B200 is different from GB200
B200 names a GPU. GB200 combines Grace CPUs and Blackwell GPUs in a system architecture. NVL72 describes a 72-GPU NVLink domain. Compare per-GPU memory separately from CPU RAM, per-tray totals and full-rack totals.
TPU versions are different from DPUs
Google publishes v5e, v5p, v6e, TPU7x (Ironwood), and announced TPU 8t and 8i. We have not verified a Google “v6p” product. A DPU such as BlueField handles infrastructure work. It does not substitute for a TPU or GPU running the model.
“S200” is ambiguous in the requested naming set. We have not assigned it specifications. Match any unfamiliar name to the manufacturer’s exact product and the provider’s instance identifier.
The GPU does not tell you the whole VM
Compare a specific instance size and placement option. Internet egress, host-to-host transport and GPU collectives can take different paths. Request topology and collective benchmark evidence for a cluster; a GPU marketplace listing alone cannot establish those properties.
Same eight H100 SXM GPUs · different network budgets
Google A3: the suffix changes the system
A3 Mega1,800 Gb/s
GPUDirect-TCPXO
A3 High1,000 Gb/s
Eight-GPU shape · GPUDirect-TCPX
A3 Edge400 Gb/s
600 Gb/s in two documented regions; otherwise 400
Browse all 34 provider configurations
| Provider / machine | Network scope | What changes the decision |
|---|---|---|
AWS G4dn / G5 / G6 / G6e ↗ | Depends on VM size; no NVLink assumption | Respectively T4, A10G, L4 and L40S. A10G is AWS’s A10-family device. Fractional and full GPU offerings are different products. |
AWS G7e ↗ | Check the selected instance size | RTX PRO 6000 Blackwell Server Edition. Do not substitute workstation power or clock specifications. |
AWS P4d / P4de ↗ | EFA; verify size and cluster placement | P4d has A100 40GB; P4de has A100 80GB. Instance RAM is separate from accelerator memory. |
AWS P5 / P5e / P5en ↗ | EFA; up to 3,200 Gb/s on P5en.48xlarge | P5 uses H100; P5e and P5en use H200. A family name alone does not specify CPU, network or GPU count. |
AWS P6-B200 ↗ | EFA; check size-specific specification | B200 server configuration. Use the current DLAMI support matrix for driver, CUDA and EFA compatibility. AWS lists 1,432 GB in the instance table and 1,440 GB in the overview for eight GPUs. Preserve this source discrepancy; measure allocatable memory inside the instance. |
AWS P6-B300 ↗ | 6,400 Gb/s EFA per eight-GPU instance | AWS lists 2,144 GB of GPU memory per instance. That differs from 8 × 288 GB nominal silicon capacity; use the actual VM budget. |
AWS P6e-GB200 ↗ | Rack-scale fabric; allocation-specific | Grace CPU architecture and the allocated NVLink domain matter to container and distributed-runtime choices. |
AWS Trn2 ↗ | NeuronLink within the system; EFA between instances | Compile for AWS Neuron. A CUDA serving command does not establish Trainium compatibility. |
Google Cloud N1 + T4 ↗ | VM network; no NVLink | An attached accelerator on an N1 machine. Confirm region, GPU quota and the host shape. |
Google Cloud G2 / G4 ↗ | VM-size-specific network | G2 uses L4; G4 uses RTX PRO 6000. Check exact GPU count and guest-visible memory. |
Google Cloud A2 Standard / A2 Ultra ↗ | VM-size-specific network | Standard uses 40GB A100; Ultra uses 80GB A100. GPU-to-GPU bandwidth is not the VM’s network bandwidth. |
Google Cloud A3 High · a3-highgpu-8g ↗ | 1,000 Gb/s maximum VM egress; GPUDirect-TCPX | Eight H100 SXM GPUs, 640 GB total HBM. Smaller High shapes have different networking and Spot/Flex-start restrictions. |
Google Cloud A3 Mega · a3-megagpu-8g ↗ | 1,800 Gb/s maximum VM egress; GPUDirect-TCPXO | Eight H100 SXM GPUs, 640 GB total HBM. Mega names a cloud system, not a new H100 chip. |
Google Cloud A3 Edge · a3-edgegpu-8g ↗ | 400 Gb/s; 600 in two documented regions | Eight H100 SXM GPUs, 640 GB total HBM; GPUDirect-TCPX. The 600 Gb/s regions are asia-south1 and northamerica-northeast2. |
Google Cloud A3 Ultra · a3-ultragpu-8g ↗ | 3,600 Gb/s maximum VM egress | Eight H200 GPUs; 1,128 GB total HBM. Reservation, Spot or Flex-start conditions apply. |
Google Cloud A4 · a4-highgpu-8g ↗ | 3,600 Gb/s maximum VM egress | Eight B200 GPUs; 1,440 GB total GPU memory, or 180 GB per GPU. |
Google Cloud A4X · a4x-highgpu-4g ↗ | 2,000 Gb/s maximum VM egress | Four GPUs, 744 GB total GPU memory, two Grace CPUs. Reserve capacity; do not add CPU RAM to HBM. |
Google Cloud A4X Max · a4x-maxgpu-4g-metal ↗ | 3,600 Gb/s maximum machine egress | Bare metal: four GPUs, 1,116 GB total GPU memory, two Grace CPUs. Capacity reservation required. |
Google Cloud TPU v4 ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Google Cloud TPU v5e ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Google Cloud TPU v5p ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Google Cloud TPU v6e ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Google Cloud TPU v7 ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Azure NCasT4_v3 / NVadsA10_v5 ↗ | Size-dependent; inspect fractional GPU allocation | T4 and A10 families respectively. The memory of a fractional GPU is smaller than a full card’s memory. |
Azure ND H100 v5 ↗ | 400 Gb/s InfiniBand per GPU; 3,200 per VM | Eight H100 GPUs. NVLink provides 900 GB/s bidirectional per GPU inside the VM; InfiniBand connects VMs. |
Azure NCads H100 v5 / ND H200 v5 ↗ | Different NC and ND system configurations | NCads H100 uses NVL; ND H200 uses H200. Confirm full size specifications before comparing them. |
Azure ND GB200 v6 / ND GB300 v6 ↗ | Check the reserved system and fabric domain | Listed in Azure’s CUDA platform catalogue. Listing does not establish quota or capacity in your subscription. |
Lambda On-demand GPU instances ↗ | Ordinary instance listing does not guarantee cluster fabric | B200 and H100 SXM come in multiple GPU counts. H100 PCIe is separately listed: choose the full product name. |
Lambda 1-Click Clusters ↗ | 400 Gb/s InfiniBand per GPU; 3,200 per node | A separate cluster product with documented rail-optimized fabric. Do not infer it from a single-GPU VM. |
CoreWeave US East region catalogue ↗ | InfiniBand appears on designated instance types | The regional catalogue identifies which configurations carry InfiniBand. Availability and reservations remain zone-specific. |
Runpod GPU type catalogue ↗ | Host-specific; require evidence for multi-node fabric | Server, workstation and Max-Q RTX PRO names differ. A marketplace GPU listing is not a topology guarantee. |
Nebius GPU VM platform catalogue ↗ | Documented H200/B200 configurations use 400 Gb/s InfiniBand NICs | Match the platform identifier and GPU count. BlueField infrastructure networking and training InfiniBand are separate paths. |
AWS Inf2 instance family ↗ | NeuronLink-v2 between chips; verify host networking for the selected size | The architecture reference documents up to twelve chips per instance. Check model partitioning and the exact size before adding chip memory together. |
AWS Trn3 architecture ↗ | NeuronLink-v4 is device-to-device; scale-out networking is a separate specification | The Neuron reference documents the chip and software architecture. It does not establish regional stock, price or an instance-level network allowance. |
Fit the weights. Then fit the workload.
What is the memory floor?
Start with total parameters, then add the memory the workload needs. This arithmetic does not certify a serving configuration. All output sizes below are decimal GB.
Override device capacity with the memory exposed by your allocation, particularly for cloud B300 and partitioned devices. The starting 32 GB budget and 15% reserve are editable teaching assumptions. They are not measurements for the selected model. Mixed-precision tensors, quantization scales, vision encoders and draft models can increase the weight payload.
320 GB
Raw weights only
320B × 8 bits ÷ 8
68 GB
Budget per device
80 GB × (1 − 15%)
6 devices
Arithmetic lower bound
round up ((weights + 32) ÷ budget)
WeightsCache + runtimeRemaining budget
This assumes perfectly balanced sharding. A result of three does not prove that a three-device parallel layout is supported. Check layer/expert divisibility, actual allocatable memory, precision kernels and the fabric before renting.
320B is the model card’s total; 18B active is not its storage size. The vLLM recipe reports about 306 GiB for the native FP8 checkpoint. Hopper requires BF16 KV for this model; the documented ROCm path targets gfx950, not every Instinct GPU. Read the model source ↗
For full training, also budget gradients, optimizer states, activations and communication buffers. The inference weight estimate above is insufficient.
Why context changes the answer
For a conventional transformer KV cache, bytes per token per sequence = 2 × layers × KV heads × head dimension × bytes per element. With 32 layers, 8 KV heads, dimension 128 and two-byte elements, that is 128 KiB per token. An 8,192-token sequence uses 1 GiB; 32 such sequences use 32 GiB.
Hybrid attention, MLA, sliding windows and recurrent states need their own accounting. Do not apply this conventional formula blindly to the recent models below.
Understand model memory →GB is not GiB
1 GB is one billion bytes. 1 GiB is 1,073,741,824 bytes. A source’s 192 GiB corresponds to about 206.16 GB. We preserve the source unit in the tables and convert to decimal GB inside the planner.
Nominal silicon capacity can also differ from memory exposed by a platform. Google lists 180 GB per B200 in A4; AWS’s P6-B300 announcement lists 2,144 GB for eight GPUs. Use the allocation you will actually receive.
Open weights: what you need to load them
A curated register for the July 12 to September 12, 2026 research window, with older baselines and undated candidates labelled separately. This is not an exhaustive release feed. Weight availability, release date and working kernels are separate checks.
Date / release scope under review
GLM-5.3-Flash ↗
Multimodal MoE · hybrid attention · MIT
320B · total parameter proxy · 18B active in the main model
320B is the model card’s total; 18B active is not its storage size. The vLLM recipe reports about 306 GiB for the native FP8 checkpoint. Hopper requires BF16 KV for this model; the documented ROCm path targets gfx950, not every Instinct GPU.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Weights verified; precise release day not certified in this reference.
2026-09-10
DeepSeek V4.1 Flash ↗
Multimodal MoE · causal encoder-decoder · MIT
552B · backbone only; conditional memory excluded · active count not used for sizing
The model card lists a 552B backbone plus 196B Engram conditional memory, with 8B active parameters in prefill and 16B in decode. Vision and draft components add further scope. Plan which tensors live in HBM, host memory and other tiers before calculating a GPU count. The generic memory screen is disabled for this preset. The reported 890 bytes per token covers compressed global KV state, not the entire runtime allocation.
Publisher announcement dated September 10 links the released weights. Date evidence ↗
2026-08-14
Qwen3.8-27B ↗
Dense multimodal · hybrid attention · Apache-2.0
27B · language-model parameter proxy; vision excluded · active count not used for sizing
27B describes the language model; the vision encoder adds memory. BF16 arithmetic gives 54 GB before that encoder, cache and runtime. A 48 GB card needs a supported lower-precision checkpoint or offload; a 16 GB card cannot hold those BF16 weights.
Author’s release_date metadata: 2026-08-14. Date evidence ↗
2026-07-27
Kimi K3 ↗
Multimodal MoE · KDA / MLA · Kimi K3 License
2,800B · total parameter proxy · 104B active in the main model
2.8T parameters imply a 1.4 TB raw 4-bit floor, before scales and non-4-bit tensors. Eight 180 GB B200s leave only 40 GB above that idealized floor. Native MXFP4 and hybrid attention require a matching serving stack; active parameters do not make this a single-card model. The vLLM recipe specifies at least eight GB300 GPUs, or eight MI355X/MI350X on ROCm; these are published configurations, not measurements reproduced here.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
vLLM’s dated day-zero post confirms public weights on July 27. Date evidence ↗
2026-08-28
GLM-5.3 ↗
Large MoE · FP8 checkpoint · MIT
753B · total parameter proxy · active count not used for sizing
753B is the Hub’s rounded tensor count, not a claim that every tensor uses FP8. The 753 GB raw 8-bit estimate excludes scales, caches and workspace. Do not reuse Flash’s architecture or serving flags for this model. The vLLM recipe describes eight H200s for FP8 and eight B200s for full context. Its 0.29.0+ header conflicts with the 0.28.0 installation; pin a working build. A 32-sequence scheduling cap does not prove 32 concurrent million-token requests. The mixed NVFP4 variant is reported at about 465 GB, larger than an all-four-bit estimate.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Official repository initial upload is labelled 0828. Date evidence ↗
2026-08-04
Shieldstral ↗
Multimodal safety classifier · Apache-2.0
3B · total parameter proxy · active count not used for sizing
A safety classifier, not a general chat replacement. The vendor describes single-16GB-GPU deployment. Six GB is only the 3B BF16 weight floor; image inputs, batch size and serving overhead still count.
Mistral’s announcement is dated August 4. Date evidence ↗
Date / release scope under review
Qwen3.8-Flash-Next ↗
Experimental MoE · n-gram embeddings · Qwen Community License 1.0
180B · total parameter proxy · 6B active in the main model
The card separates 125B model parameters, 51B n-gram embeddings and a 4B draft module. This planner includes all three. Six active billion refers to routed model computation, not total memory. Eight-bit storage here is a hypothetical comparison, not a certified checkpoint recipe.
Weights verified; release date remains under review.
Date / release scope under review
Hunyuan Hy4-preview ↗
Preview MoE · sparse attention · See publisher’s model license
780B · total parameter proxy · 49B active in the main model
770B backbone plus 10B draft parameters gives 780B for the planner. The draft adds active computation too. An advertised million-token context is a model limit, not a promise that the full context fits at useful concurrency. Eight-bit storage is illustrative here.
Preview weights verified; release date remains under review.
Date / release scope under review
DeepSeek V4 Pro ↗
MoE · hybrid sparse attention · See publisher’s model license
1,600B · total parameter proxy · 49B active in the main model
The checkpoint mixes FP4 experts with higher-precision tensors. The 800 GB all-4-bit floor understates real loading memory. Do not treat an API model name or release date as the exact downloadable checkpoint revision.
Comparison baseline; an API refresh is not evidence of new weights in this window.
Choose for the job you will actually run
Learning, embeddings, small-model serving
Start with the smallest single device that fits your model and useful batch size. T4, L4 and A10 are worth checking for supported smaller workloads. L40S and RTX PRO 6000 offer larger single-device memory. Compare the actual hourly quote and measured latency.
Move up when cache pressure, unsupported precision or latency makes the smaller device a poor fit. Buying more peak compute alone does not fix those problems.
Large-model inference
Fit weights, then budget context and concurrent requests. H200, B200, B300 and Instinct’s larger-memory parts can reduce sharding pressure. A smaller GPU count is useful only if the runtime supports that layout and the remaining memory holds the workload.
Long-context serving may benefit more from memory capacity than a higher compute peak. Check a real recipe before choosing a brand-new architecture.
Full training across devices
Budget optimizer states and activations as well as weights. One common mixed-precision AdamW accounting is 2 bytes of weights + 2 of gradients + 4 of master weights + 8 of optimizer moments: 16 bytes per parameter before activations.
A 7B model then has about 112 GB of model states. Sharding can distribute those states; it also adds communication. HGX-class systems, compatible Instinct clusters and TPU slices deserve evaluation when collectives are central to the workload.
Fine-tuning with adapters
LoRA trains a smaller set of parameters, reducing gradient and optimizer memory. The base model still has to be loaded, and activations still grow with sequence length and batch size. Quantized base weights can help if the training stack supports them.
Reverse the single-device choice when a longer sequence or a larger base model no longer fits. Establish correctness on the intended precision before scaling the run.
A fast first token and fast streaming are different tests
A request can wait in a queue before any GPU work begins. Long prompts then add prefill time. Once the first token arrives, decoding repeats until the response ends. Report each part, because an aggregate tokens-per-second number hides the experience of one person waiting.
One request · intervals are illustrative
Queue + routing
Waiting, authentication and scheduling
Prefill → first token
Process prompt; sample the first output token
Decode → decode → …
Generate subsequent tokens and stream them
Make a latency claim reproducible
Record checkpoint revision, quantization, serving-engine commit, GPU type and count, tensor/expert parallelism, interconnect, prompt/output lengths, concurrency or arrival rate, prefix-cache hit rate and speculative decoding settings.
Report p50 and p95 TTFT, TPOT or inter-token latency with its definition, end-to-end latency, output tokens/s per replica, and failures. Compare systems at the same latency target.
A sanity check you can do on paper
If a dense 7B BF16 model streams 14 GB of weights once per decode step at a peak 3.35 TB/s, weight reads alone take at least 14 ÷ 3,350 seconds, about 4.18 ms. Real execution adds other work and rarely reaches peak bandwidth.
This assumes batch one and a weight-streaming workload. It is not a measured H100 TPOT, and it does not apply unchanged to MoE, cached weights or batched serving.
Learn the roofline model →Compare cost at the latency you need
Use the full replica’s hourly cost and measured output tokens per second that meet your latency target. Cost per million output tokens = hourly replica cost × 1,000,000 ÷ (3,600 × qualifying output tokens/s).
For an illustrative $12/hour replica delivering 200 qualifying output tokens/s, the result is $16.67 per million output tokens. This is arithmetic, not a provider quote or benchmark. Include idle time, failed requests and required host/network costs in the measurement and bill.
Record region, currency, purchase term, accelerator count and quote date. A low hourly price alone cannot rank providers.
Measured performance coverage: no model × hardware benchmark runs are published in this reference yet. We link recipes and show explicit memory arithmetic; we do not turn peak TFLOPS into promised tokens per second.
Use this in an interview
“I would first check the total weight payload, the runtime’s supported precision, and the memory needed at our target context and concurrency. Then I would pick a feasible parallel layout and check the links it uses. Finally, I would benchmark TTFT and streaming latency at our arrival rate before comparing cost.”
Sources and review dates appear on every accelerator profile. Announced hardware is labelled separately. No price, stock or benchmark number is inferred from a product name.
Cite this reference
AI Infra Interviews. GPU and TPU hardware comparison. Reviewed . https://aiinfrainterviews.com/hardware
Include your access date when citing a changing specification. Link to the specification section for hardware figures or the explanation for a sizing or workload decision.
Research method and limits
We compile manufacturer specifications, cloud documentation, model cards and serving recipes. The reference preserves source units and distinguishes individual devices from nodes and racks. Conflicting figures and unknown fields remain labelled.
Our contribution is the comparison, unit reconciliation, worked arithmetic and workload explanation. Published peaks are vendor specifications. Calculator results are estimates under the displayed assumptions. Neither is a measurement from our own accelerator lab.
For a manufacturer’s specification, consult the sources linked from each hardware profile. Cite our page when using its analysis, and retain primary-source attribution for underlying figures. This curated reference does not establish market share, live availability or a universal performance ranking.
Found a discrepancy? Send a correction with the page URL, exact variant, disputed figure and supporting primary document.