H100 SXM, PCIe and NVL
SXM is a module used in tightly connected server platforms. H100 PCIe is a different 80 GB configuration. H100 NVL has 94 GB per GPU; a bridged pair has 188 GB combined. Memory bandwidth, power and connectivity differ.
Hardware field guide · free reference · source dates on each profile
GPU & TPU hardware comparison
An H100 is a chip. “A3 Mega” describes a cloud machine around it. Learn which specifications change your model’s memory budget, which links carry its tensors, and what to check before choosing a GPU or TPU.
New to the names? Start with the visual GPU map: desktop cards, server GPUs and connected racks, with a memory exercise and a guide to the generations.
Follow the bytes · conceptual topology
Three bandwidths, three different jobs
Local memoryWeights + KV cache + activations
Compute enginesExecute kernels on these bytes
① Memory bandwidth: inside each accelerator
Accelerator AOwn local memory
Accelerator BOwn local memory
② Scale-up: NVLink, Infinity Fabric or PCIe
Moves sharded tensors and collective results between devices
Moves sharded tensors and collective results between devices
Server AAccelerators + host
Server BAnother fabric domain
③ Scale-out: NICs + switches + placement
InfiniBand, RoCE, EFA or provider-specific transport
InfiniBand, RoCE, EFA or provider-specific transport
Start with memory
Will it load with room to work?
A 70B model at two bytes per parameter needs about 140 GB for weights alone. Two 80 GB GPUs provide 160 GB nominal capacity, leaving 20 GB before runtime, quantization metadata and request state. That is a budget to investigate, not a deployment guarantee.
Work through your model →Then follow the traffic
Where will the bytes move?
Local memory bandwidth helps feed each accelerator. Links between accelerators move sharded tensors. The cluster network connects servers. A strong number on one of those paths cannot compensate for a bottleneck on another.
Compare the cloud configurations →Browse 37 accelerator profiles. Specs come from linked manufacturer and provider documents. Workload choices are engineering guidance; timing results require a benchmark.
New to the hardware vocabulary? Start with these labels.
- HBM / GDDR
- Memory attached to the accelerator. Capacity tells you how much fits; bandwidth tells you how quickly bytes can move. HBM means high-bandwidth memory; GDDR is graphics double data rate memory.
- BF16 / FP8 / FP4
- Number formats using roughly 16, 8 or 4 bits per value. Fewer bits can reduce storage and work, but the exact format, scales, kernels and model accuracy still matter.
- KV cache
- The stored keys and values used by attention to reuse earlier token computations. More active requests or longer contexts often need more cache.
- NVLink / NVSwitch
- NVIDIA’s links between accelerators and the switches used to connect larger groups. A bridge between two cards and a switched server are different topologies.
- ICI / DCN
- Google’s inter-chip interconnect connects TPU chips within a slice. The data-center network connects separate slices.
- InfiniBand / RoCE / EFA
- Cluster communication technologies. RoCE carries remote direct memory access over Ethernet; EFA is AWS’s Elastic Fabric Adapter. A NIC is the network interface that joins a server to its network.
- Dense / sparse
- Dense compute figures do not assume skipped weights. Structured-sparse peaks depend on a supported sparsity pattern. Keep those two kinds of peak separate.
- TTFT / TPOT
- Time to first token measures the initial wait. Time per output token describes subsequent generation, with an explicit averaging definition.
01 · Hardware specifications
Compare the exact variant
Open a profile for precision-labelled compute, power, partitioning, interconnect and workload tradeoffs. Consumer cards are included for orientation; server hardware is the focus.
37 of 37 accelerators · select up to four to compare
| Compare | Accelerator | Memory | Memory bandwidth | Useful starting point |
|---|---|---|---|---|
| NVIDIA T4 Turing | 16 GB GDDR6 | 0.32 TB/s | A small inference GPU for a model that already fits. Its 16 GB capacity is usually the first limit a new LLM user encounters. Documented product | |
| NVIDIA A10 Ampere | 24 GB GDDR6 | 0.6 TB/s | A 24 GB graphics-capable server GPU. It offers more memory than T4, without the HBM or switched links of A100. Documented product | |
| NVIDIA L4 Ada Lovelace | 24 GB GDDR6 | 0.3 TB/s | A low-power 24 GB inference and media GPU. More compute does not imply more memory bandwidth: its published 300 GB/s is below A10. Documented product | |
| NVIDIA L40S Ada Lovelace | 48 GB GDDR6 | 0.86 TB/s | A 48 GB server card for inference, fine-tuning and graphics. The extra capacity is useful; it does not bring an HGX interconnect. Documented product | |
| NVIDIA A100 40 GB SXM Ampere | 40 GB HBM2 | 1.56 TB/s | 40 GB of HBM on the SXM4 form factor. A100 remains useful for BF16 training and supported inference, but cannot execute Hopper’s native FP8 Tensor Core path. Documented product | |
| NVIDIA A100 80 GB SXM Ampere | 80 GB HBM2e | 2.04 TB/s | 80 GB of HBM on the SXM4 form factor. A100 remains useful for BF16 training and supported inference, but cannot execute Hopper’s native FP8 Tensor Core path. Documented product | |
| NVIDIA A100 80 GB PCIe Ampere | 80 GB HBM2e | 1.94 TB/s | 80 GB of HBM on the PCIe form factor. A100 remains useful for BF16 training and supported inference, but cannot execute Hopper’s native FP8 Tensor Core path. Documented product | |
| NVIDIA H100 80 GB SXM Hopper | 80 GB HBM3 | 3.35 TB/s | The common eight-GPU training-node H100. Its fast local GPU fabric is separate from whatever network connects the node to another node. Documented product | |
| NVIDIA H100 80 GB PCIe Hopper | 80 GB HBM2e | 2 TB/s | The 80 GB PCIe H100 is a different operating point from the SXM card. The same H100 name does not promise 3.35 TB/s memory bandwidth. Documented product | |
| NVIDIA H100 NVL Hopper | 94 GB HBM3 | 3.9 TB/s | H100 NVL is a 94 GB PCIe GPU often discussed as a two-card, 188 GB pair. Always count how many cards the listing includes. Documented product | |
| NVIDIA H200 SXM Hopper | 141 GB HBM3e | 4.8 TB/s | Hopper with a larger, faster memory system: 141 GB at 4.8 TB/s. Its value over H100 is easiest to see in memory capacity and memory traffic. Documented product | |
| NVIDIA B200 HGX Blackwell | 180 GB HBM3e | 8 TB/s | An HBM-rich Blackwell GPU for large training and inference jobs. Use the 180 GB HGX usable-memory specification, not an early 192 GB announcement. Documented product | |
| NVIDIA B300 HGX Blackwell Ultra | 288 GB HBM3e | 8 TB/s | Blackwell Ultra’s larger HBM budget helps large models and long context. A cloud allocation may expose less memory than the 288 GB physical-product figure. Documented product | |
| NVIDIA GB200 NVL72 Grace Blackwell | 186 GB HBM3e | 8 TB/s | A rack-scale system, not a single PCIe card. This comparison uses 186 GB per GPU from GCP’s four-GPU VM configuration; the full rack includes 72 GPUs and 36 Grace CPUs. Documented product | |
| NVIDIA GB300 NVL72 Grace Blackwell Ultra | 279 GB HBM3e | 8 TB/s | A rack-scale system, not a single PCIe card. This comparison uses 279 GB per GPU from GCP’s four-GPU VM configuration; the full rack includes 72 GPUs and 36 Grace CPUs. Documented product | |
| RTX PRO 6000 Blackwell Server Edition Blackwell | 96 GB GDDR7 | 1.6 TB/s | 96 GB of GDDR7 makes this an interesting inference and mixed AI/graphics card. Its memory capacity sits above H100, but its bandwidth does not. Documented product | |
| AMD Instinct MI300X CDNA 3 | 192 GB HBM3 | 5.3 TB/s | 192 GB of HBM on an AMD CDNA 3 accelerator. The memory is useful only after the model’s kernels and serving engine work on the ROCm target. Documented product | |
| AMD Instinct MI325X CDNA 3 | 256 GB HBM3e | 6 TB/s | 256 GB of HBM on an AMD CDNA 3 accelerator. The memory is useful only after the model’s kernels and serving engine work on the ROCm target. Documented product | |
| AMD Instinct MI355X CDNA 4 | 288 GB HBM3e | 8 TB/s | 288 GB of HBM on an AMD CDNA 4 accelerator. The memory is useful only after the model’s kernels and serving engine work on the ROCm target. Documented product | |
| Google TPU v4 TPU v4 | 32 GiB HBM | 1.2 TB/s | A mature training-oriented TPU with 32 GiB per chip. Existing XLA jobs can be a better starting point than porting a CUDA-only workload. Documented product | |
| Google TPU v5e TPU v5e | 16 GB HBM | 0.86 TB/s | The smaller-memory TPU branch for economical training and serving. Eight chips provide eight 16 GB budgets, not one shared 128 GB allocation. Documented product | |
| Google TPU v5p TPU v5p | 95 GiB HBM | 2.77 TB/s | A large-pod training TPU with 95 GiB per chip, much more memory than v5e. The suffix identifies a different design, not a larger v5e VM. Documented product | |
| Google TPU v6e (Trillium) TPU v6e (Trillium) | 32 GB HBM | 1.64 TB/s | Trillium doubles v5e’s nominal memory to 32 GB and increases BF16 compute. It retains a 256-chip, 2D pod design. Documented product | |
| Google TPU7x (Ironwood) TPU7x (Ironwood) | 192 GiB HBM | 7.38 TB/s | Ironwood adds a much larger HBM budget and native FP8. Google’s technical name is TPU7x; its two chiplets are visible as two framework devices per chip. Documented product | |
| Google TPU 8t TPU 8t | 216 GB HBM | 6.53 TB/s | Google’s announced eighth-generation training design. It emphasizes large-scale pre-training, a 3D torus and a new scale-out network. Announced; orderability not verified | |
| Google TPU 8i TPU 8i | 288 GB HBM | 8.6 TB/s | Google’s announced eighth-generation serving design. Larger SRAM and Boardfly target communication and state movement in reasoning workloads. Announced; orderability not verified | |
| AWS Trainium2 NeuronCore-v3 | 96 GiB HBM | 2.9 TB/s | AWS’s own training accelerator, also used for supported inference. It runs the Neuron stack rather than a CUDA binary. Documented product | |
| GeForce RTX 5090 Blackwell | 32 GB GDDR7 | 1.79 TB/s | A 32 GB desktop card for local learning and supported small-model inference. The memory limit remains real even when the compute headline looks large. Documented product | |
| GeForce RTX 5080 Blackwell | 16 GB GDDR7 | 0.96 TB/s | A 16 GB desktop card for local learning and supported small-model inference. The memory limit remains real even when the compute headline looks large. Documented product | |
| GeForce RTX 5070 Blackwell | 12 GB GDDR7 | 0.67 TB/s | A 12 GB desktop card for local learning and supported small-model inference. The memory limit remains real even when the compute headline looks large. Documented product | |
| NVIDIA BlueField-3 DPU Infrastructure offload | Not a model-memory device | N/A | A DPU moves infrastructure work such as networking, security and storage off the host. It is not another name for Google TPU. Documented product | |
| AWS Inferentia2 NeuronCore-v2 | 32 GiB HBM | 0.88 TB/s | An inference-focused AWS chip with two NeuronCore-v2 cores. Inf2 instances can contain up to twelve chips. Documented product | |
| AWS Trainium3 NeuronCore-v4 | 144 GiB HBM | 4.9 TB/s | Eight NeuronCore-v4 cores and 144 GiB of HBM per chip. Its low-precision and memory upgrades are much larger than its dense BF16 compute increase over Trainium2. Documented product | |
| NVIDIA A30 Ampere | 24 GB HBM2 | 0.93 TB/s | A 24 GB HBM2 accelerator for shared enterprise servers. It has more memory bandwidth than L4, but an older compute architecture. Documented product | |
| NVIDIA H200 NVL Hopper | 141 GB HBM3e | 4.8 TB/s | NVIDIA H200 NVL is a dual-slot PCIe GPU with 141 GB HBM3e, 4.8 TB/s memory bandwidth and up to 600 W configurable power. Its dense BF16/FP16 peak is 835.5 TFLOPS; compare its bridge topology and compute limits with H200 SXM below. Documented product | |
| RTX PRO 6000 Blackwell Workstation Edition Blackwell | 96 GB GDDR7 ECC | 1.79 TB/s | A desktop card with 96 GB of memory for local model work and graphics. Its 1.792 TB/s specification differs from the Server Edition’s 1.597 TB/s. Documented product | |
| AMD Instinct MI350X CDNA 4 | 288 GB HBM3e | 8 TB/s | MI350X and MI355X share 288 GB of HBM3e and 8 TB/s peak bandwidth. MI350X has a lower board-power specification and lower compute peaks. Documented product |
H200 NVL vs H200 SXM: what changes?
Both variants provide 141 GB of HBM3e and 4.8 TB/s of memory bandwidth per GPU. NVL is a PCIe card with a lower compute and power ceiling; SXM is a module used in HGX systems. Choose the server topology as well as the GPU.
| Specification | H200 NVL | H200 SXM |
|---|---|---|
| Memory per GPU | 141 GB HBM3e | 141 GB HBM3e |
| Memory bandwidth | 4.8 TB/s | 4.8 TB/s |
| Dense tensor compute | 835.5 TFLOPS BF16/FP16; 1,670.5 TFLOPS FP8, dense | 989.5 TFLOPS BF16/FP16; 1,979 TFLOPS FP8, dense |
| Form factor | Dual-slot, air-cooled PCIe card | SXM module |
| Power limit | Up to 600 W, configurable | Up to 700 W |
| Scale-up links | Two- or four-way NVLink bridge: 900 GB/s per GPU | NVLink 4: 900 GB/s bidirectional per GPU |
Why do H200 NVL BF16 figures say both 835.5 and 1,671 TFLOPS?
NVIDIA labels its 1,671 TFLOPS BF16/FP16 figure “with sparsity.” The dense equivalent is half: 835.5 TFLOPS. Its 3,341 TFLOPS FP8 sparse peak similarly corresponds to 1,670.5 TFLOPS dense. Compare the same precision and sparsity convention; ordinary dense weights do not earn the sparse rate automatically.
NVIDIA H200 specification table and footnotes ↗, checked October 21, 2026 UTC. NVIDIA still labels the table preliminary. Confirm the purchased server’s power setting and two- or four-GPU NVLink bridge layout; eight cards in a chassis do not establish an eight-GPU NVSwitch domain.
To turn these numbers into a decision, work through decode arithmetic intensity and the accelerator selection guide, then benchmark the model at its target context and concurrency.
The names that cause expensive mix-ups
Mega, High and Edge are cloud machines
Google’s eight-GPU A3 High, Mega and Edge machines all use H100 SXM 80 GB. Their documented maximum VM network rates differ: 1,000, 1,800 and 400–600 Gb/s. Those labels do not create new H100 silicon.
Google’s machine specifications ↗B200 is different from GB200
B200 names a GPU. GB200 combines Grace CPUs and Blackwell GPUs in a system architecture. NVL72 describes a 72-GPU NVLink domain. Compare per-GPU memory separately from CPU RAM, per-tray totals and full-rack totals.
TPU versions are different from DPUs
Google publishes v5e, v5p, v6e, TPU7x (Ironwood), and announced TPU 8t and 8i. We have not verified a Google “v6p” product. A DPU such as BlueField handles infrastructure work. It does not substitute for a TPU or GPU running the model.
“S200” is ambiguous in the requested naming set. We have not assigned it specifications. Match any unfamiliar name to the manufacturer’s exact product and the provider’s instance identifier.
The GPU does not tell you the whole VM
Compare a specific instance size and placement option. Internet egress, host-to-host transport and GPU collectives can take different paths. Request topology and collective benchmark evidence for a cluster; a GPU marketplace listing alone cannot establish those properties.
Same eight H100 SXM GPUs · different network budgets
Google A3: the suffix changes the system
A3 Mega1,800 Gb/s
GPUDirect-TCPXO
A3 High1,000 Gb/s
Eight-GPU shape · GPUDirect-TCPX
A3 Edge400 Gb/s
600 Gb/s in two documented regions; otherwise 400
Browse all 34 provider configurations
| Provider / machine | Network scope | What changes the decision |
|---|---|---|
AWS G4dn / G5 / G6 / G6e ↗ | Depends on VM size; no NVLink assumption | Respectively T4, A10G, L4 and L40S. A10G is AWS’s A10-family device. Fractional and full GPU offerings are different products. |
AWS G7e ↗ | Check the selected instance size | RTX PRO 6000 Blackwell Server Edition. Do not substitute workstation power or clock specifications. |
AWS P4d / P4de ↗ | EFA; verify size and cluster placement | P4d has A100 40GB; P4de has A100 80GB. Instance RAM is separate from accelerator memory. |
AWS P5 / P5e / P5en ↗ | EFA; up to 3,200 Gb/s on P5en.48xlarge | P5 uses H100; P5e and P5en use H200. A family name alone does not specify CPU, network or GPU count. |
AWS P6-B200 ↗ | EFA; check size-specific specification | B200 server configuration. Use the current DLAMI support matrix for driver, CUDA and EFA compatibility. AWS lists 1,432 GB in the instance table and 1,440 GB in the overview for eight GPUs. Preserve this source discrepancy; measure allocatable memory inside the instance. |
AWS P6-B300 ↗ | 6,400 Gb/s EFA per eight-GPU instance | AWS lists 2,144 GB of GPU memory per instance. That differs from 8 × 288 GB nominal silicon capacity; use the actual VM budget. |
AWS P6e-GB200 ↗ | Rack-scale fabric; allocation-specific | Grace CPU architecture and the allocated NVLink domain matter to container and distributed-runtime choices. |
AWS Trn2 ↗ | NeuronLink within the system; EFA between instances | Compile for AWS Neuron. A CUDA serving command does not establish Trainium compatibility. |
Google Cloud N1 + T4 ↗ | VM network; no NVLink | An attached accelerator on an N1 machine. Confirm region, GPU quota and the host shape. |
Google Cloud G2 / G4 ↗ | VM-size-specific network | G2 uses L4; G4 uses RTX PRO 6000. Check exact GPU count and guest-visible memory. |
Google Cloud A2 Standard / A2 Ultra ↗ | VM-size-specific network | Standard uses 40GB A100; Ultra uses 80GB A100. GPU-to-GPU bandwidth is not the VM’s network bandwidth. |
Google Cloud A3 High · a3-highgpu-8g ↗ | 1,000 Gb/s maximum VM egress; GPUDirect-TCPX | Eight H100 SXM GPUs, 640 GB total HBM. Smaller High shapes have different networking and Spot/Flex-start restrictions. |
Google Cloud A3 Mega · a3-megagpu-8g ↗ | 1,800 Gb/s maximum VM egress; GPUDirect-TCPXO | Eight H100 SXM GPUs, 640 GB total HBM. Mega names a cloud system, not a new H100 chip. |
Google Cloud A3 Edge · a3-edgegpu-8g ↗ | 400 Gb/s; 600 in two documented regions | Eight H100 SXM GPUs, 640 GB total HBM; GPUDirect-TCPX. The 600 Gb/s regions are asia-south1 and northamerica-northeast2. |
Google Cloud A3 Ultra · a3-ultragpu-8g ↗ | 3,600 Gb/s maximum VM egress | Eight H200 GPUs; 1,128 GB total HBM. Reservation, Spot or Flex-start conditions apply. |
Google Cloud A4 · a4-highgpu-8g ↗ | 3,600 Gb/s maximum VM egress | Eight B200 GPUs; 1,440 GB total GPU memory, or 180 GB per GPU. |
Google Cloud A4X · a4x-highgpu-4g ↗ | 2,000 Gb/s maximum VM egress | Four GPUs, 744 GB total GPU memory, two Grace CPUs. Reserve capacity; do not add CPU RAM to HBM. |
Google Cloud A4X Max · a4x-maxgpu-4g-metal ↗ | 3,600 Gb/s maximum machine egress | Bare metal: four GPUs, 1,116 GB total GPU memory, two Grace CPUs. Capacity reservation required. |
Google Cloud TPU v4 ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Google Cloud TPU v5e ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Google Cloud TPU v5p ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Google Cloud TPU v6e ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Google Cloud TPU v7 ↗ | ICI within a slice; DCN between slices | Select a documented slice shape and supported region. Chip, chiplet, TensorCore, host and VM counts are not interchangeable. |
Azure NCasT4_v3 / NVadsA10_v5 ↗ | Size-dependent; inspect fractional GPU allocation | T4 and A10 families respectively. The memory of a fractional GPU is smaller than a full card’s memory. |
Azure ND H100 v5 ↗ | 400 Gb/s InfiniBand per GPU; 3,200 per VM | Eight H100 GPUs. NVLink provides 900 GB/s bidirectional per GPU inside the VM; InfiniBand connects VMs. |
Azure NCads H100 v5 / ND H200 v5 ↗ | Different NC and ND system configurations | NCads H100 uses NVL; ND H200 uses H200. Confirm full size specifications before comparing them. |
Azure ND GB200 v6 / ND GB300 v6 ↗ | Check the reserved system and fabric domain | Listed in Azure’s CUDA platform catalogue. Listing does not establish quota or capacity in your subscription. |
Lambda On-demand GPU instances ↗ | Ordinary instance listing does not guarantee cluster fabric | B200 and H100 SXM come in multiple GPU counts. H100 PCIe is separately listed: choose the full product name. |
Lambda 1-Click Clusters ↗ | 400 Gb/s InfiniBand per GPU; 3,200 per node | A separate cluster product with documented rail-optimized fabric. Do not infer it from a single-GPU VM. |
CoreWeave US East region catalogue ↗ | InfiniBand appears on designated instance types | The regional catalogue identifies which configurations carry InfiniBand. Availability and reservations remain zone-specific. |
Runpod GPU type catalogue ↗ | Host-specific; require evidence for multi-node fabric | Server, workstation and Max-Q RTX PRO names differ. A marketplace GPU listing is not a topology guarantee. |
Nebius GPU VM platform catalogue ↗ | Documented H200/B200 configurations use 400 Gb/s InfiniBand NICs | Match the platform identifier and GPU count. BlueField infrastructure networking and training InfiniBand are separate paths. |
AWS Inf2 instance family ↗ | NeuronLink-v2 between chips; verify host networking for the selected size | The architecture reference documents up to twelve chips per instance. Check model partitioning and the exact size before adding chip memory together. |
AWS Trn3 architecture ↗ | NeuronLink-v4 is device-to-device; scale-out networking is a separate specification | The Neuron reference documents the chip and software architecture. It does not establish regional stock, price or an instance-level network allowance. |
Fit the weights. Then fit the workload.
Start here: smaller checkpoints
Try an 8B checkpoint before renting a multi-GPU machine. You can learn the same loading, cache and batching decisions with a smaller memory bill. Choose a model whose licence and runtime support your task; the table below screens weight size only.
| Checkpoint | Weight storage | Evidence |
|---|---|---|
| Llama 3.1 8B Instruct | 16.06 GB | Pinned public metadata ↗ |
| Qwen3 8B | 16.38 GB | Pinned tensor index ↗ |
| Qwen3 30B-A3B | 61.06 GB | Pinned tensor index ↗ |
| Gemma 3 27B IT | 54.86 GB | Pinned public metadata ↗ |
| Gemma 4 E2B IT | 10.25 GB | Pinned public metadata ↗ |
| Gemma 4 E4B IT | 15.99 GB | Pinned public metadata ↗ |
| Gemma 4 12B IT | 23.92 GB | Pinned public metadata ↗ |
| Gemma 4 26B-A4B IT | 51.61 GB | Pinned tensor index ↗ |
| Gemma 4 31B IT | 62.55 GB | Pinned tensor index ↗ |
| Qwen3 32B | 65.52 GB | Pinned tensor index ↗ |
| GLM-4.7 Flash | 62.44 GB | Pinned public metadata ↗ |
Gemma 4 E2B stores about 5.12 billion BF16 tensor elements, roughly 10.25 GB. Its effective parameter label describes computation. Qwen3-30B-A3B activates about 3.3B parameters per token but stores about 30.53B; inactive experts still need a placement plan.
Worked example · Qwen3-8B · BF16 weights and cache
One conversation fits this budget. Two do not.
16.38 GB weights
From the pinned checkpoint index.
+ 1.21 GB per conversation
8,192 cached tokens, including generated tokens.
+ 2.00 GB runtime
An illustrative allowance, to replace with a measurement.
A nominal 24 GB L4 with 15% left unallocated gives a 20.40 GB budget. One sequence totals 19.59 GB. Two total 20.80 GB, over that budget.
The calculator starts with that example’s rounded 3.21 GB cache-plus-runtime allowance. Replace it for your workload. Gemma’s mixed attention layers and cache sharing need their own accounting; the Qwen formula does not apply unchanged.
Learn model memory · Work through KV cache sizing · Start the foundations course
What is the memory floor?
Start with total parameters, then add the memory the workload needs. This arithmetic does not certify a serving configuration. All output sizes below are decimal GB.
Override device capacity with the memory exposed by your allocation, particularly for cloud B300 and partitioned devices. The starting 3.21 GB budget and 15% reserve are editable teaching assumptions. They are not measurements for the selected model. Mixed-precision tensors, quantization scales, vision encoders and draft models can increase the weight payload.
16.38 GB
Raw weights only
8.19073536B × 16 bits ÷ 8
20.4 GB
Budget per device
24 GB × (1 − 15%)
1 devices
Arithmetic lower bound
round up ((weights + 3.21) ÷ budget)
WeightsCache + runtimeRemaining budget
This assumes perfectly balanced sharding. A result of three does not prove that a three-device parallel layout is supported. Check layer/expert divisibility, actual allocatable memory, precision kernels and the fabric before renting.
The pinned BF16 tensor index reports 16.38 GB before cache and runtime. Its 36 layers and eight KV heads use 144 KiB per retained token with BF16 cache. The card documents 32,768 native context and YaRN extension; the config’s 40,960 value is not a separate quality guarantee. Thinking tokens consume the output budget too. Read the model source ↗
For full training, also budget gradients, optimizer states, activations and communication buffers. The inference weight estimate above is insufficient.
Why context changes the answer
For a conventional transformer KV cache, bytes per token per sequence = 2 × layers × KV heads × head dimension × bytes per element. With 32 layers, 8 KV heads, dimension 128 and two-byte elements, that is 128 KiB per token. An 8,192-token sequence uses 1 GiB; 32 such sequences use 32 GiB.
Hybrid attention, MLA, sliding windows and recurrent states need their own accounting. Do not apply this conventional formula blindly to the recent models below.
Understand model memory →GB is not GiB
1 GB is one billion bytes. 1 GiB is 1,073,741,824 bytes. A source’s 192 GiB corresponds to about 206.16 GB. We preserve the source unit in the tables and convert to decimal GB inside the planner.
Nominal silicon capacity can also differ from memory exposed by a platform. Google lists 180 GB per B200 in A4; AWS’s P6-B300 announcement lists 2,144 GB for eight GPUs. Use the allocation you will actually receive.
Open weights: what you need to load them
Start with a smaller checkpoint to learn memory sizing, then compare larger models. Sources were checked September 12, 2026; older baselines and uncertain release dates are labelled. Weight availability, release date and working kernels are separate checks.
Llama 3.1 8B Instruct · 8.03B · Dense text · grouped-query attention
2024-07-23
Llama 3.1 8B Instruct ↗
Dense text · grouped-query attention · Llama 3.1 Community License
8.03B · stored BF16 tensor-element proxy; not an active or non-embedding count · active count not used for sizing
A useful first text-serving model. The public Hub metadata lists about 8.03B BF16 tensor elements, giving a 16.06 GB weight estimate. Cache and runtime still need space. Weight downloads require accepting Meta’s access terms; config and tensor-index downloads were gated during this review.
Checkpoint evidence ↗ · revision 0e9e39f2
Public Hub BF16 element count; weight bytes are estimated, not an inspected tensor-index total.
Publisher card dates the Llama 3.1 release to July 23, 2024; retained as a small-model baseline. Date evidence ↗
Qwen3 8B · 8.19B · Dense text · grouped-query attention
Date / release scope under review
Qwen3 8B ↗
Dense text · grouped-query attention · Apache 2.0
8.19B · stored BF16 tensor-element proxy; not an active or non-embedding count · active count not used for sizing
The pinned BF16 tensor index reports 16.38 GB before cache and runtime. Its 36 layers and eight KV heads use 144 KiB per retained token with BF16 cache. The card documents 32,768 native context and YaRN extension; the config’s 40,960 value is not a separate quality guarantee. Thinking tokens consume the output budget too.
Checkpoint evidence ↗ · revision b968826d
Tensor-index total_size; excludes file headers, KV cache and runtime allocation.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint and architecture scope reviewed on September 12, 2026. No exact release-day claim is made.
Qwen3 30B-A3B · 30.53B · Text MoE · grouped-query attention
Date / release scope under review
Qwen3 30B-A3B ↗
Text MoE · grouped-query attention · Apache 2.0
30.53B · stored BF16 tensor-element proxy; not an active or non-embedding count · 3.3B active in the main model
The A3B name refers to active computation. The BF16 checkpoint contains 61.06 GB of tensors, so sizing it as a 3B model is wrong. Its 48 layers and four KV heads give a conventional BF16 KV payload of 96 KiB per token, before runtime allocation. A lower-bit checkpoint needs its own compatible kernels and quality check.
Checkpoint evidence ↗ · revision ad44e777
Tensor-index total_size; excludes file headers, KV cache and runtime allocation.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint and architecture scope reviewed on September 12, 2026. No exact release-day claim is made.
Gemma 3 27B IT · 27.43B · Multimodal dense · local/global attention
Date / release scope under review
Gemma 3 27B IT ↗
Multimodal dense · local/global attention · Gemma terms
27.43B · stored BF16 tensor-element proxy; not an active or non-embedding count · active count not used for sizing
The public Hub lists about 27.43B BF16 tensor elements, including more than a rounded text-model label captures. That implies about 54.86 GB for raw tensors. Image processing and mixed local/global attention add workload-specific state. The card is public, but config and tensor-index downloads require accepting Google’s access terms; no exact index-byte claim is made here.
Checkpoint evidence ↗ · revision 005ad340
Public Hub BF16 element count; weight bytes are estimated, not an inspected tensor-index total.
Checkpoint and architecture scope reviewed on September 12, 2026. No exact release-day claim is made.
Gemma 4 E2B IT · 5.12B · Multimodal dense · per-layer embeddings
Date / release scope under review
Gemma 4 E2B IT ↗
Multimodal dense · per-layer embeddings · Apache 2.0
5.12B · stored BF16 tensor-element proxy; not an active or non-embedding count · active count not used for sizing
E2B describes effective parameters, not all stored weights. The pinned Hub metadata lists about 5.12B BF16 tensor elements, implying 10.25 GB before cache and runtime. This includes large per-layer embedding tables; it is not a 4 GB BF16 checkpoint. Audio/vision inputs and an optional draft model need separate budgets. Use a Gemma 4-capable engine.
Checkpoint evidence ↗ · revision 3e22461f
Public Hub BF16 element count; weight bytes are estimated, not an inspected tensor-index total.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint and architecture scope reviewed on September 12, 2026. No exact release-day claim is made.
Gemma 4 E4B IT · 8B · Multimodal dense · per-layer embeddings
Date / release scope under review
Gemma 4 E4B IT ↗
Multimodal dense · per-layer embeddings · Apache 2.0
8B · stored BF16 tensor-element proxy; not an active or non-embedding count · active count not used for sizing
E4B is an effective-compute label. The pinned Hub metadata lists about 8.00B BF16 tensor elements, implying 15.99 GB before cache and runtime. Per-layer embeddings explain why the full payload is larger than four billion weights. Its local/global attention needs architecture-aware cache accounting; an ordinary all-layers KV formula is insufficient.
Checkpoint evidence ↗ · revision ee0ef602
Public Hub BF16 element count; weight bytes are estimated, not an inspected tensor-index total.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint and architecture scope reviewed on September 12, 2026. No exact release-day claim is made.
Gemma 4 12B IT · 11.96B · Unified multimodal dense · local/global attention
Date / release scope under review
Gemma 4 12B IT ↗
Unified multimodal dense · local/global attention · Apache 2.0
11.96B · stored BF16 tensor-element proxy; not an active or non-embedding count · active count not used for sizing
The pinned Hub metadata lists about 11.96B BF16 tensor elements, implying 23.92 GB before cache and runtime. A nominal 24 GB accelerator leaves almost no room beyond that floor. The unified architecture has different global and local attention settings; do not copy E2B cache estimates or assume an older Gemma runtime can load it.
Checkpoint evidence ↗ · revision 707f0a3b
Public Hub BF16 element count; weight bytes are estimated, not an inspected tensor-index total.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint and architecture scope reviewed on September 12, 2026. No exact release-day claim is made.
Gemma 4 26B-A4B IT · 25.81B · Multimodal MoE · local/global attention
Date / release scope under review
Gemma 4 26B-A4B IT ↗
Multimodal MoE · local/global attention · Apache 2.0
25.81B · stored BF16 tensor-element proxy; not an active or non-embedding count · 3.8B active in the main model
The BF16 index reports 51.61 GB of tensor payload. The card’s active 3.8B figure describes per-token work, not storage. Index total_parameters and stored BF16 element counts differ; this preset uses stored elements for weight arithmetic. Local and global attention have different head dimensions and sharing rules, so derive cache from this exact config.
Checkpoint evidence ↗ · revision 4d7ae498
Tensor-index total_size; excludes file headers, KV cache and runtime allocation.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint and architecture scope reviewed on September 12, 2026. No exact release-day claim is made.
Gemma 4 31B IT · 31.27B · Multimodal dense · local/global attention
Date / release scope under review
Gemma 4 31B IT ↗
Multimodal dense · local/global attention · Apache 2.0
31.27B · stored BF16 tensor-element proxy; not an active or non-embedding count · active count not used for sizing
The BF16 index reports 62.55 GB of tensor payload. The architectural label and stored-element count differ; the preset uses the latter for arithmetic. The 60-layer config mixes local and global attention with different head dimensions and KV-head counts. Vision state, cache allocation and a separate speculative draft model are outside the weight payload.
Checkpoint evidence ↗ · revision 842da379
Tensor-index total_size; excludes file headers, KV cache and runtime allocation.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint and architecture scope reviewed on September 12, 2026. No exact release-day claim is made.
Qwen3 32B · 32.76B · Dense text · grouped-query attention
Date / release scope under review
Qwen3 32B ↗
Dense text · grouped-query attention · Apache 2.0
32.76B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · active count not used for sizing
The pinned BF16 index contains 65.52 GB of tensors. Its 64 layers, eight KV heads and 128-element heads require 256 KiB per cached token with BF16 cache: 2 GiB for one 8192-token sequence, before allocation overhead. The card documents 32768 native context with a separate YaRN extension; the config value alone is not a quality guarantee.
Checkpoint evidence ↗ · revision 9216db57
BF16 index bytes agree with the public tensor count; cache and runtime are additional.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
GLM-4.7 Flash · 31.22B · Text MoE · latent attention
Date / release scope under review
GLM-4.7 Flash ↗
Text MoE · latent attention · MIT
31.22B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · active count not used for sizing
Public metadata lists about 31.22B elements, mostly BF16, implying about 62.44 GB before cache. The index total_size matches the element count rather than BF16 bytes; do not quote it as a 31 GB checkpoint. Latent attention uses kv_lora_rank 512 and separate positional state. Check the selected engine’s cache representation and the pinned card’s version-specific serving instructions.
Checkpoint evidence ↗ · revision 7dd20894
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
GLM-5.3-Flash · 320B · Multimodal MoE · hybrid attention
Date / release scope under review
GLM-5.3-Flash ↗
Multimodal MoE · hybrid attention · MIT
320B · total parameter proxy · 18B active in the main model
320B is the model card’s total; 18B active is not its storage size. The vLLM recipe reports about 306 GiB for the native FP8 checkpoint. Hopper requires BF16 KV for this model; the documented ROCm path targets gfx950, not every Instinct GPU.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Weights verified; precise release day not certified in this reference.
DeepSeek V4.1 Flash · 552B · Multimodal MoE · causal encoder-decoder
2026-09-10
DeepSeek V4.1 Flash ↗
Multimodal MoE · causal encoder-decoder · MIT
552B · backbone only; conditional memory excluded · active count not used for sizing
The model card lists a 552B backbone plus 196B Engram conditional memory, with 8B active parameters in prefill and 16B in decode. Vision and draft components add further scope. Plan which tensors live in HBM, host memory and other tiers before calculating a GPU count. The generic memory screen is disabled for this preset. The reported 890 bytes per token covers compressed global KV state, not the entire runtime allocation.
Publisher announcement dated September 10 links the released weights. Date evidence ↗
Qwen3.8-27B · 27B · Dense multimodal · hybrid attention
2026-08-14
Qwen3.8-27B ↗
Dense multimodal · hybrid attention · Apache-2.0
27B · language-model parameter proxy; vision excluded · active count not used for sizing
27B describes the language model; the vision encoder adds memory. BF16 arithmetic gives 54 GB before that encoder, cache and runtime. A 48 GB card needs a supported lower-precision checkpoint or offload; a 16 GB card cannot hold those BF16 weights.
Author’s release_date metadata: 2026-08-14. Date evidence ↗
Kimi K3 · 2,800B · Multimodal MoE · KDA / MLA
2026-07-27
Kimi K3 ↗
Multimodal MoE · KDA / MLA · Kimi K3 License
2,800B · total parameter proxy · 104B active in the main model
2.8T parameters imply a 1.4 TB raw 4-bit floor, before scales and non-4-bit tensors. Eight 180 GB B200s leave only 40 GB above that idealized floor. Native MXFP4 and hybrid attention require a matching serving stack; active parameters do not make this a single-card model. The vLLM recipe specifies at least eight GB300 GPUs, or eight MI355X/MI350X on ROCm; these are published configurations, not measurements reproduced here.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
vLLM’s dated day-zero post confirms public weights on July 27. Date evidence ↗
GLM-5.3 · 753B · Large MoE · FP8 checkpoint
2026-08-28
GLM-5.3 ↗
Large MoE · FP8 checkpoint · MIT
753B · total parameter proxy · active count not used for sizing
753B is the Hub’s rounded tensor count, not a claim that every tensor uses FP8. The 753 GB raw 8-bit estimate excludes scales, caches and workspace. Do not reuse Flash’s architecture or serving flags for this model. The vLLM recipe describes eight H200s for FP8 and eight B200s for full context. Its 0.29.0+ header conflicts with the 0.28.0 installation; pin a working build. A 32-sequence scheduling cap does not prove 32 concurrent million-token requests. The mixed NVFP4 variant is reported at about 465 GB, larger than an all-four-bit estimate.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Official repository initial upload is labelled 0828. Date evidence ↗
Shieldstral · 3B · Multimodal safety classifier
2026-08-04
Shieldstral ↗
Multimodal safety classifier · Apache-2.0
3B · total parameter proxy · active count not used for sizing
A safety classifier, not a general chat replacement. The vendor describes single-16GB-GPU deployment. Six GB is only the 3B BF16 weight floor; image inputs, batch size and serving overhead still count.
Mistral’s announcement is dated August 4. Date evidence ↗
Qwen3.8-Flash-Next · 180B · Experimental MoE · n-gram embeddings
Date / release scope under review
Qwen3.8-Flash-Next ↗
Experimental MoE · n-gram embeddings · Qwen Community License 1.0
180B · total parameter proxy · 6B active in the main model
The card separates 125B model parameters, 51B n-gram embeddings and a 4B draft module. This planner includes all three. Six active billion refers to routed model computation, not total memory. Eight-bit storage here is a hypothetical comparison, not a certified checkpoint recipe.
Weights verified; release date remains under review.
Hunyuan Hy4-preview · 780B · Preview MoE · sparse attention
Date / release scope under review
Hunyuan Hy4-preview ↗
Preview MoE · sparse attention · See publisher’s model license
780B · total parameter proxy · 49B active in the main model
770B backbone plus 10B draft parameters gives 780B for the planner. The draft adds active computation too. An advertised million-token context is a model limit, not a promise that the full context fits at useful concurrency. Eight-bit storage is illustrative here.
Preview weights verified; release date remains under review.
DeepSeek V4 Pro · 1,600B · MoE · hybrid sparse attention
Date / release scope under review
DeepSeek V4 Pro ↗
MoE · hybrid sparse attention · See publisher’s model license
1,600B · total parameter proxy · 49B active in the main model
The checkpoint mixes FP4 experts with higher-precision tensors. The 800 GB all-4-bit floor understates real loading memory. Do not treat an API model name or release date as the exact downloadable checkpoint revision.
Comparison baseline; an API refresh is not evidence of new weights in this window.
Llama 3.3 70B Instruct · 70.55B · Dense text
Date / release scope under review
Llama 3.3 70B Instruct ↗
Dense text · Llama 3.3 Community License
70.55B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · active count not used for sizing
Public BF16 metadata implies about 141.11 GB of weights. Moving from an 8B model to this baseline changes the sharding problem before any cache is allocated. Config and index requests were gated during review; verify your downloaded checkpoint and quantized variant before choosing a layout.
Checkpoint evidence ↗ · revision 6f6073b4
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
Llama 3.1 405B Instruct · 405.85B · Dense text
Date / release scope under review
Llama 3.1 405B Instruct ↗
Dense text · Llama 3.1 Community License
405.85B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · active count not used for sizing
Public BF16 metadata implies about 811.71 GB of weights. The 405B name is rounded; use the downloaded tensor payload for a close fit. Config and index requests were gated. A smaller precision requires an actual converted checkpoint and compatible kernels, not just changing the calculator to four bits.
Checkpoint evidence ↗ · revision be673f32
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
Llama 4 Scout 17B-16E Instruct · 108.64B · Multimodal MoE
Date / release scope under review
Llama 4 Scout 17B-16E Instruct ↗
Multimodal MoE · Llama 4 Community License
108.64B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · 17B active in the main model
The 17B label describes active work. Public metadata lists about 108.64B BF16 elements, roughly 217.28 GB of weights. The vision path and long-context cache require additional memory. Config/index access was gated, so this reference does not certify a context length, quantized fit or engine layout.
Checkpoint evidence ↗ · revision 92f3b159
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
Llama 4 Maverick 17B-128E Instruct · 401.58B · Multimodal MoE
Date / release scope under review
Llama 4 Maverick 17B-128E Instruct ↗
Multimodal MoE · Llama 4 Community License
401.58B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · 17B active in the main model
Scout and Maverick both say 17B active, but Maverick stores about 401.58B BF16 elements in public metadata, roughly 803.17 GB. Expert count changes storage and communication even when active compute looks similar. Gated config/index access prevented exact file-level verification here.
Checkpoint evidence ↗ · revision 73d14711
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
Qwen3 235B-A22B · 235.09B · Text MoE · grouped-query attention
Date / release scope under review
Qwen3 235B-A22B ↗
Text MoE · grouped-query attention · Apache 2.0
235.09B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · 22B active in the main model
The BF16 tensor index contains 470.19 GB, despite only about 22B parameters being active per token. Its 94 layers and four KV heads still accumulate cache. Budget expert storage and communication separately from active FLOPs; a memory-only GPU count does not validate expert or tensor parallel divisibility.
Checkpoint evidence ↗ · revision 8efa6172
BF16 index bytes agree with the public tensor count; cache and runtime are additional.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
Qwen3.5 397B-A17B · 403.4B · Multimodal MoE · hybrid attention
Date / release scope under review
Qwen3.5 397B-A17B ↗
Multimodal MoE · hybrid attention · Apache 2.0
403.4B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · 17B active in the main model
The architectural label is 397B total and 17B active; public checkpoint metadata includes about 403.40B elements and a small FP32 component. The index reports 806.80 GB. Hybrid attention state and vision processing need their own runtime accounting. Do not multiply every layer by the conventional KV formula or size the weights from 17B.
Checkpoint evidence ↗ · revision 84726181
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
GLM-4.5 Air · 110.47B · Text MoE · grouped-query attention
Date / release scope under review
GLM-4.5 Air ↗
Text MoE · grouped-query attention · MIT
110.47B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · 12B active in the main model
The card describes 106B total and 12B active; packaged metadata lists about 110.47B elements, mostly BF16. Its index total_size equals the element count, so it is not accepted here as byte evidence. A BF16 estimate is about 220.94 GB before cache. Check the downloaded tensor headers and the runtime treatment of next-token prediction layers.
Checkpoint evidence ↗ · revision a24ceef6
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
GLM-5 · 753.86B · Text MoE · sparse latent attention
Date / release scope under review
GLM-5 ↗
Text MoE · sparse latent attention · MIT
753.86B · public stored-element proxy; hypothetical uniform precision excludes dtype overhead · 40B active in the main model
The publisher describes 744B main-model parameters and 40B active; packaged metadata lists about 753.86B elements. Sparse attention reduces selected attention work, not all weight storage. The 78-layer config includes latent attention state and next-token prediction configuration, so runtime memory must be derived from the exact loaded modules.
Checkpoint evidence ↗ · revision c183ef8c
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
Kimi K2.5 · 1,000B · Multimodal MoE · packed expert weights
Date / release scope under review
Kimi K2.5 ↗
Multimodal MoE · packed expert weights · Modified MIT
1,000B · publisher main-model count; packed checkpoint needs separate byte accounting · 32B active in the main model
The card describes a 1T model with 32B active and a vision encoder. Its checkpoint uses packed four-bit expert tensors alongside unquantized modules. Integer container counts are not a parameter-to-byte conversion; the index reports about 595.15 GB. Confirm tensor packing, scales and loaded modules before sizing. Latent attention and images add workload-specific state.
Checkpoint evidence ↗ · revision 4d01dfe0
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
DeepSeek V3 · 671B · Text MoE · FP8 and latent attention
Date / release scope under review
DeepSeek V3 ↗
Text MoE · FP8 and latent attention · DeepSeek Model License; code MIT
671B · publisher main-model count; packed checkpoint needs separate byte accounting · 37B active in the main model
The card separates a 671B main model from roughly 14B of multi-token prediction weights. Public metadata mixes FP8, BF16 and FP32, while index size metadata follows a different accounting. Neither 671B × one byte nor a blind index total is a certified loaded payload. Check which prediction layers the engine loads and its FP8 conversion path.
Checkpoint evidence ↗ · revision e815299b
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
DeepSeek R1 · 671B · Text MoE · FP8 and latent attention
Date / release scope under review
DeepSeek R1 ↗
Text MoE · FP8 and latent attention · MIT
671B · publisher main-model count; packed checkpoint needs separate byte accounting · 37B active in the main model
This is the full R1 checkpoint, not one of the smaller distilled models. Its mixed FP8/BF16 metadata and main-model count have different scopes; inspect the exact loaded payload before deriving devices. Long reasoning outputs consume cache and serving time. Cache allocation follows the latent-attention implementation rather than the ordinary full-head KV formula.
Checkpoint evidence ↗ · revision 56d4cbbb
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
DeepSeek V4 Flash · 284B · Text MoE · compressed attention and mixed precision
Date / release scope under review
DeepSeek V4 Flash ↗
Text MoE · compressed attention and mixed precision · MIT
284B · publisher main-model count; packed checkpoint needs separate byte accounting · 13B active in the main model
The card labels 284B total and 13B active. Expert FP4 and other FP8 tensors form a mixed checkpoint; the index reports about 159.61 GB, which differs from uniform four-bit arithmetic. Compressed attention requires architecture-specific state accounting. A million-token context claim does not establish a practical batch size or latency target.
Checkpoint evidence ↗ · revision 60d8d707
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
GPT-OSS 120B · 117B · Text MoE · MXFP4 experts and hybrid attention
Date / release scope under review
GPT-OSS 120B ↗
Text MoE · MXFP4 experts and hybrid attention · Apache 2.0
117B · publisher main-model count; packed checkpoint needs separate byte accounting · 5.1B active in the main model
The card describes 117B parameters with 5.1B active. MXFP4 experts coexist with other tensor types; its index reports about 65.25 GB, above the 58.5 GB obtained from uniform four-bit arithmetic. The publisher’s single-80-GB-GPU example is not a guarantee at arbitrary context or concurrency. Verify MXFP4 kernels and alternating sliding/full attention cache allocation.
Checkpoint evidence ↗ · revision b5c939de
Public checkpoint metadata; mixed precision, gated access or conflicting index metadata prevents an exact uniform-byte claim.
Published serving recipe ↗
Compatibility evidence; no local throughput or latency measurement.
Checkpoint scope reviewed September 12, 2026; no exact release-day claim is made.
Choose for the job you will actually run
Learning, embeddings, small-model serving
Start with the smallest single device that fits your model and useful batch size. T4, L4 and A10 are worth checking for supported smaller workloads. L40S and RTX PRO 6000 offer larger single-device memory. Compare the actual hourly quote and measured latency.
Move up when cache pressure, unsupported precision or latency makes the smaller device a poor fit. Buying more peak compute alone does not fix those problems.
Large-model inference
Fit weights, then budget context and concurrent requests. H200, B200, B300 and Instinct’s larger-memory parts can reduce sharding pressure. A smaller GPU count is useful only if the runtime supports that layout and the remaining memory holds the workload.
Long-context serving may benefit more from memory capacity than a higher compute peak. Check a real recipe before choosing a brand-new architecture.
Full training across devices
Budget optimizer states and activations as well as weights. One common mixed-precision AdamW accounting is 2 bytes of weights + 2 of gradients + 4 of master weights + 8 of optimizer moments: 16 bytes per parameter before activations.
A 7B model then has about 112 GB of model states. Sharding can distribute those states; it also adds communication. HGX-class systems, compatible Instinct clusters and TPU slices deserve evaluation when collectives are central to the workload.
Fine-tuning with adapters
LoRA trains a smaller set of parameters, reducing gradient and optimizer memory. The base model still has to be loaded, and activations still grow with sequence length and batch size. Quantized base weights can help if the training stack supports them.
Reverse the single-device choice when a longer sequence or a larger base model no longer fits. Establish correctness on the intended precision before scaling the run.
A fast first token and fast streaming are different tests
A request can wait in a queue before any GPU work begins. Long prompts then add prefill time. Once the first token arrives, decoding repeats until the response ends. Report each part, because an aggregate tokens-per-second number hides the experience of one person waiting.
One request · intervals are illustrative
Queue + routing
Waiting, authentication and scheduling
Prefill → first token
Process prompt; sample the first output token
Decode → decode → …
Generate subsequent tokens and stream them
Make a latency claim reproducible
Record checkpoint revision, quantization, serving-engine commit, GPU type and count, tensor/expert parallelism, interconnect, prompt/output lengths, concurrency or arrival rate, prefix-cache hit rate and speculative decoding settings.
Report p50 and p95 TTFT, TPOT or inter-token latency with its definition, end-to-end latency, output tokens/s per replica, and failures. Compare systems at the same latency target.
A sanity check you can do on paper
If a dense 7B BF16 model streams 14 GB of weights once per decode step at a peak 3.35 TB/s, weight reads alone take at least 14 ÷ 3,350 seconds, about 4.18 ms. Real execution adds other work and rarely reaches peak bandwidth.
This assumes batch one and a weight-streaming workload. It is not a measured H100 TPOT, and it does not apply unchanged to MoE, cached weights or batched serving.
Learn the roofline model →Compare cost at the latency you need
Use the full replica’s hourly cost and measured output tokens per second that meet your latency target. Cost per million output tokens = hourly replica cost × 1,000,000 ÷ (3,600 × qualifying output tokens/s).
For an illustrative $12/hour replica delivering 200 qualifying output tokens/s, the result is $16.67 per million output tokens. This is arithmetic, not a provider quote or benchmark. Include idle time, failed requests and required host/network costs in the measurement and bill.
Record region, currency, purchase term, accelerator count and quote date. A low hourly price alone cannot rank providers.
Measured performance coverage: no model × hardware benchmark runs are published in this reference yet. We link recipes and show explicit memory arithmetic; we do not turn peak TFLOPS into promised tokens per second.
Use this in an interview
“I would first check the total weight payload, the runtime’s supported precision, and the memory needed at our target context and concurrency. Then I would pick a feasible parallel layout and check the links it uses. Finally, I would benchmark TTFT and streaming latency at our arrival rate before comparing cost.”
Sources and review dates appear on every accelerator profile. Announced hardware is labelled separately. No price, stock or benchmark number is inferred from a product name.
Cite this reference
AI Infra Interviews. GPU and TPU hardware comparison. Reviewed . https://aiinfrainterviews.com/hardware
Include your access date when citing a changing specification. Link to the specification section for hardware figures or the explanation for a sizing or workload decision.
Research method and limits
We compile manufacturer specifications, cloud documentation, model cards and serving recipes. The reference preserves source units and distinguishes individual devices from nodes and racks. Conflicting figures and unknown fields remain labelled.
Our contribution is the comparison, unit reconciliation, worked arithmetic and workload explanation. Published peaks are vendor specifications. Calculator results are estimates under the displayed assumptions. Neither is a measurement from our own accelerator lab.
For a manufacturer’s specification, consult the sources linked from each hardware profile. Cite our page when using its analysis, and retain primary-source attribution for underlying figures. This curated reference does not establish market share, live availability or a universal performance ranking.
Found a discrepancy? Send a correction with the page URL, exact variant, disputed figure and supporting primary document.