First model comparison
Qwen3.5-4B
A small dense multimodal example. Read its size, input support and memory screen before considering a larger model.
Start here / No GPU background needed
Your first open model.
Start with the basics.
You have found a model called “35B-A3B” and a download labeled “4-bit”. Before deciding whether to use it, let’s make those labels mean something.
01 / The model itself
What are model weights? What makes them open?
A model stores learned numbers called weights. During training, those numbers are adjusted so the model gets better at its training objective. During inference, the model uses them to process your input and produce an output.
An open-weight model makes its trained parameters available to download. Depending on its license, you may be able to run it yourself, adapt it or redistribute a version. You can also use a provider that hosts those weights for you; downloading them is optional.
Architecture
The arrangement of layers and connections.
Weights
The learned values used inside that arrangement.
Runtime
The software that loads the model and performs the computation.
Application
Your interface, tools, data and rules around the model.
Two models can share an architecture and have different weights, just as the same program can load different saved data. That does not mean their answers or quality will be the same.
Weights alone do not necessarily include the code and data information needed to reproduce training. See the Open Source AI Definition for the broader open-source claim.
02 / Read the label
What do 8B, 35B-A3B and “effective” mean?
A parameter is a learned numerical value in the model. B means billion: an 8B model has roughly eight billion parameters. A larger count can give a model more capacity, but it does not guarantee better results on your task.
| Label | Read it as | What it does not mean |
|---|---|---|
| 8B dense | About 8 billion stored parameters. Its feed-forward computation does not route among a sparse pool of experts. | 8 GB of memory, or a guarantee of quality. |
| 35B-A3B MoE | About 35 billion total, with about 3 billion active for a token. MoE means mixture of experts: a router selects part of the expert pool. | Only 3 billion parameters need storage. |
| Gemma E2B | The effective count describes a smaller computation path; the full count includes additional embedding tables. | A conventional 2B-sized download. |
A token is a unit of input or output used by the model. It may be a word, part of a word or another encoded unit. The tokenizer decides the split, so word count and token count are different.
03 / Make the memory visible
Quantization stores numbers with less precision.
Using fewer bits per value can make the model smaller. A bit is a unit of information; eight bits make one byte. Common 16-bit formats such as BF16 and FP16 use two bytes per value. A uniform 8-bit representation uses one; a uniform 4-bit representation uses half a byte.
8 billion numbers. Three ways to store them.
16-bit
16 GB
8-bit
8 GB
4-bit
4 GB
Illustrative uniform-weight arithmetic: billions of parameters × bits ÷ 8 = decimal GB. Bars share a zero baseline and are proportional to stored bytes. Real formats add scales or other metadata, and can mix precisions.
Less precision uses less memory. Whether it preserves enough quality, and whether it runs faster, must be tested with the actual model and runtime.
Quantization approximates values. It is not the same as zipping a file and later recovering every original value unchanged. The model may still perform well, but you need to test the exact conversion on the tasks you care about.
04 / Leave room for the conversation
The weights can fit while the request runs out of memory.
GPU memory, often called VRAM, holds more than model weights. The running request also needs state. A KV cache retains attention information so the model can reuse previous work; some models keep other kinds of state as well.
The context window is the sequence budget the model or deployment accepts. Longer inputs, longer outputs and more simultaneous requests can require more memory. A large advertised context is not a promise that your hardware can serve it quickly.
A small example
An illustrative 8B model has 16 GB of uniform 16-bit weights. That does not establish that a 16 GB GPU can run it: there is no room left for request state or runtime buffers. Four-bit weights have a 4 GB arithmetic floor, but the actual conversion and remaining allocations still matter.
05 / Choose a small, useful task
You do not need to buy a GPU to start.
- Pick a task you can check
Try extracting a few fields, classifying a message, or generating a small function with tests. Decide what a correct result looks like.
- Choose where to run it
A hosted endpoint lets you evaluate without owning hardware. A supported local runtime lets you try a model on a machine you already have. Check the exact artifact and device support first.
- Keep a small-model baseline
Compare a larger model only when you can name what the small one gets wrong. More parameters or a newer release may not fix the actual problem.
- Record the complete setup
Keep the model revision, quantization, runtime and prompts. Test your real inputs, tool calls and failure cases before relying on the output.
Local inference can keep the model computation on your machine. Your application may still use external search, tools or logging, so check the complete data flow.
A different storage format
gpt-oss-20b
A text-reasoning example with mixed MXFP4 storage. The publisher’s 16 GB memory class is a starting point, not a guarantee for every device.
Check that the labels now make sense.
A 35B-A3B model activates about 3B parameters for a token. Should you budget stored weights for 3B or 35B?
Show the explanation
Start with the full 35B store. Routing reduces the work selected for a token; it does not remove the other experts. Then account for the actual precision and runtime state.
Once that distinction feels comfortable, you are ready for the model cards. The hardware recipes come next; the detailed attention architecture can wait until you need to explain its costs.
Keep learning, one step at a time.
Beginner: the terms and worked examples on this page. Intermediate: model selection, hardware and memory budgets. Advanced: attention layouts, reproducible deployment and the courses.
The model reference is public. Linked course access varies. Technical examples use the pinned sources on the linked model cards; arithmetic is illustrative, not a deployment certification. Reviewed 13 September 2026.