AI Infra Interviews logo

Nemotron / Mamba-2 / attention / latent MoE

Nemotron 3.5 Bolt 74B-A9B 24GB

A 24 GB checkpoint still needs headroom

Open-weight releaseSources checked 2026-11-16

A useful candidate when local multimodal deployment needs stronger compression. Treat its packed representation as a kernel requirement as well as a storage saving.

74Breported total parameters
9Bactive parameters per token
262,144configured token positions

Publisher-rounded multimodal count. The 88 language blocks include 40 Mamba, 40 MoE and eight attention blocks. Routed expert counts vary from 240 to 496 by block.

Evaluate it for

Local vision-language agent evaluation with an explicitly supported Bolt runtime.

Choose another path when

Only 24 GB of total available memory is assumed sufficient, or the runtime cannot execute this mixed packing.

Release context

Why this release matters

Compression changes per-block expert counts, and the quantization config mixes formats. The global 512-expert default does not describe each block in this exported architecture.

The publisher card explicitly dates the Hugging Face release.

Release evidence · Follow the Nemotron timeline →

Deployment starting points

Which hardware, how many, at what precision?

Start with an exact artifact and an engine that supports it. A configuration below is evidence of a documented path; its device count does not promise a particular throughput or concurrency.

No certified hardware configuration in this edition

Only 24 GB of total available memory is assumed sufficient, or the runtime cannot execute this mixed packing.

The pinned model card establishes the architecture. It does not, by itself, prove that a particular GPU count runs this checkpoint at your target context. The capacity screen below helps rule out undersized allocations; a successful load and workload test are still required.

24.08 GB of tensor data in the pinned index

This is the index’s reported tensor payload, in decimal GB. It excludes file headers and runtime memory. Mixed precision, conversion and host/device placement determine how much GPU memory the loaded model needs.

Inspect the exact index metadata →
Capacity screen: would the weights fit?

This estimates inference memory for a hypothetical uniform precision. It is useful for rejecting an allocation that is too small. It does not establish a working deployment or estimate training memory.

A uniform-precision GPU count would hide this model’s component layout. Publisher-rounded multimodal count. The 88 language blocks include 40 Mamba, 40 MoE and eight attention blocks. Routed expert counts vary from 240 to 496 by block. Use the documented artifact and placement path above; no GPU count is inferred here.

Architecture in practice

What the serving engine has to do

Two kinds of layers retain two kinds of state.

A token enters a layerthe layer type selects itspathRecurrent layerupdate a compact stateAttention layerread retained tokenhistoryRecurrent statemaintained for thissequenceAttention cachelayout depends on themodel

Hybrid attention does not mean zero request state.

A full-attention formula cannot describe every layer.

Schematic of a hybrid attention stack, not a literal layer ordering. Recurrent layers update a compact state; attention layers retain their own history. Both kinds of state matter for prefix reuse and concurrent requests.

Mechanism schematic based on the pinned model card and configuration. It explains a design principle; it is not a full implementation graph.

Inspect the full checkpoint specifications
Inspected fields for this exact revision. KV heads alone do not describe latent, hybrid or shared-cache layouts.
Exact checkpointnvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB
Text model typenemotron_h_puzzle
Layers88
Hidden width4,096
Attention / KV heads32 / 2
Head dimension128
Routed / selected expertsNot verified / 9
Configured positions262,144
Documented extensionNot recorded for this checkpoint
Inputs → outputtext, image, video → text
License metadataOpenMDW-1.1 · custom terms

40 Mamba layers; 40 MoE layers; 8 attention layers. These counts describe the configured pattern.

Open weights do not imply unrestricted use. Read the applicable license terms linked from the pinned card.

A useful comparison

What else belongs on the shortlist?

Super VL is the less-compressed parent comparison; the Liquid alternative is text-only. Hold modality and task requirements constant when measuring cost.

13 Oct 2026

LFM2.5-24B-A2B

Active parameters are a compute clue, not a device budget

24B reported total · text

These are editorial comparison candidates. We have not run a matched quality or serving benchmark, so this is not a ranking.

Prepare to explain it

The interview lesson

Read block_configs before reporting one expert count. Then separate the target artifact, optional drafter and per-request state.

Attention layouts · Quantization · Courses and worked examples

For a deployment evaluation, record exact weights, precision, engine, device count, interconnect, prompt/output lengths and concurrency. Report task success, errors, TTFT, TPOT and useful throughput together.

Sources and citation

Our assessment is editorial judgment based on the linked architecture and deployment evidence. Model facts are publisher-reported or attributed to runtime maintainers; no independent GPU benchmark was run.

AI Infra Interviews, “Nemotron 3.5 Bolt 74B-A9B 24GB: release, architecture and deployment”, checked 2026-11-16. Preserve this date and the exact checkpoint when citing.

Model and research notes as JSON · NVIDIA profile