AI Infra Interviews logo

Nemotron / Mamba-2 / attention / latent MoE

Nemotron 3.5 Super VL 120B-A12B NVFP4

Follow the mixed-precision recipe all the way to runtime state

Open-weight releaseSources checked 2026-11-16

A multimodal agent candidate where a documented NVIDIA deployment is useful. The pinned recipe provides a starting configuration; it is not our throughput measurement.

120Breported total parameters
12Bactive parameters per token
1,048,576configured token positions

Publisher-reported rounded multimodal count. The 88 configured language blocks are 40 Mamba, 40 MoE and eight attention blocks; vision and MTP are separate.

Evaluate it for

Text, document-image and sampled-video evaluation on supported NVIDIA hardware.

Choose another path when

A generic backend without the required Mamba or quantization support is mandatory.

Release context

Why this release matters

The configuration separates forty Mamba blocks, forty expert blocks and eight attention blocks. Mamba cache precision and restoration are part of the serving contract, not properties inferred from the weight dtype.

The publisher card explicitly dates the Hugging Face release.

Release evidence · Follow the Nemotron timeline →

Deployment starting points

Which hardware, how many, at what precision?

Start with an exact artifact and an engine that supports it. A configuration below is evidence of a documented path; its device count does not promise a particular throughput or concurrency.

Runtime recipe

1 × H200 SXM

141 GB per GPU

Mixed NVFP4; non-Blackwell activation path

vLLM, publisher model-specific setup

Publisher adapts the two-H100 command to one H200 with TP1. Maximum 128 sequences, three-token MTP and float16 Mamba state with stochastic rounding; up to 1M context does not mean 128 simultaneous 1M requests.

Checkpoint and source

nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4

Checked 2026-11-16. Read the documented setup →

Hardware specifications and interconnect →

75.01 GB of tensor data in the pinned index

This is the index’s reported tensor payload, in decimal GB. It excludes file headers and runtime memory. Mixed precision, conversion and host/device placement determine how much GPU memory the loaded model needs.

Inspect the exact index metadata →
Capacity screen: would the weights fit?

This estimates inference memory for a hypothetical uniform precision. It is useful for rejecting an allocation that is too small. It does not establish a working deployment or estimate training memory.

A uniform-precision GPU count would hide this model’s component layout. Publisher-reported rounded multimodal count. The 88 configured language blocks are 40 Mamba, 40 MoE and eight attention blocks; vision and MTP are separate. Use the documented artifact and placement path above; no GPU count is inferred here.

Architecture in practice

What the serving engine has to do

Two kinds of layers retain two kinds of state.

A token enters a layerthe layer type selects itspathRecurrent layerupdate a compact stateAttention layerread retained tokenhistoryRecurrent statemaintained for thissequenceAttention cachelayout depends on themodel

Hybrid attention does not mean zero request state.

A full-attention formula cannot describe every layer.

Schematic of a hybrid attention stack, not a literal layer ordering. Recurrent layers update a compact state; attention layers retain their own history. Both kinds of state matter for prefix reuse and concurrent requests.

Mechanism schematic based on the pinned model card and configuration. It explains a design principle; it is not a full implementation graph.

Inspect the full checkpoint specifications
Inspected fields for this exact revision. KV heads alone do not describe latent, hybrid or shared-cache layouts.
Exact checkpointnvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4
Text model typenemotron_h
Layers88
Hidden width4,096
Attention / KV heads32 / 2
Head dimension128
Routed / selected experts512 / 22
Configured positions1,048,576
Documented extensionNot recorded for this checkpoint
Inputs → outputtext, image, video → text
License metadataOpenMDW-1.1 · custom terms

40 Mamba layers; 40 MoE layers; 8 attention layers. These counts describe the configured pattern.

Open weights do not imply unrestricted use. Read the applicable license terms linked from the pinned card.

A useful comparison

What else belongs on the shortlist?

Bolt compresses this family further. Compare the actual task accuracy, precision path and cache allocation rather than only total parameters.

16 Apr 2026

Qwen3.6-35B-A3B

A smaller expert store for coding-agent experiments

35B reported total · text + image + video

These are editorial comparison candidates. We have not run a matched quality or serving benchmark, so this is not a ranking.

Prepare to explain it

The interview lesson

Explain why 120B times half a byte understates this 75.01 GB artifact, then list the runtime allocations still missing.

Attention layouts · Quantization · Courses and worked examples

For a deployment evaluation, record exact weights, precision, engine, device count, interconnect, prompt/output lengths and concurrency. Report task success, errors, TTFT, TPOT and useful throughput together.

Sources and citation

Our assessment is editorial judgment based on the linked architecture and deployment evidence. Model facts are publisher-reported or attributed to runtime maintainers; no independent GPU benchmark was run.

AI Infra Interviews, “Nemotron 3.5 Super VL 120B-A12B NVFP4: release, architecture and deployment”, checked 2026-11-16. Preserve this date and the exact checkpoint when citing.

Model and research notes as JSON · NVIDIA profile