TL;DR: Read metadata before downloading weights: the exact revision, license, architecture, tensor formats and a supported engine recipe. Derive a workload-specific, per-rank memory estimate with explicit assumptions. Report a candidate configuration and what could invalidate it. Configuration fields alone do not prove runtime memory, throughput, or a delivery date.
How to approach it
A product manager asks whether a newly announced model can be evaluated this week. Establish the task-quality requirement and prompt/output envelope. Check whether an existing deployment path supports the artifact on available hardware. A missing optimized kernel might require a slower reference path or engineering work; it does not justify inventing a completion date.
A strong answer
Record the repository revision before reading its metadata. Open-Weights Models of 2026 shows why this matters: updated checkpoints can add speculative modules, and a hosted API may offer features absent from the downloadable artifact.
| Read first | Question it answers |
|---|---|
| Card, intended use and license | Is this artifact a candidate for the task and intended deployment? |
| Architecture, nested text/vision config and layer types | Which implementation and attention/state formulas apply? |
| Tensor index, dtypes, scales and quantization exclusions | What is stored, including auxiliary tensors? |
| Engine recipe, pinned build and hardware path | Can it load and execute the required combination? |
| Tokenizer/template and context configuration | What does the request contract actually support? |
Start with the small metadata files so you can decide whether transferring the weights is worthwhile. Reading configuration does not require executing repository code. If the loader does require custom code, review it and pin the revision before running it. The Hugging Face custom-model guidance describes that distinction.
Now work a synthetic release with explicitly supplied properties: 1.2T weights, exactly 90% in MXFP4 and 10% in BF16, 88 latent-attention layers, 576 latent values plus 64 rotary values, and a BF16 cache. Assume the engine retains one full latent-cache copy on each tensor-parallel rank.
stored weight estimate, complete groups and no padding
1.2e12 × 0.90 × 0.53125 = 573.75 GB
1.2e12 × 0.10 × 2 = 240.00 GB
total 813.75 GB
one logical cache component per position
88 × (576 + 64) × 2 = 112,640 B = 110 KiB
product envelope: 32 sequences × 8,192 positions
one cache copy = 29,527,900,160 B = 29.528 GB
The weight split is a premise of this exercise. For a real release, obtain it from tensor metadata instead of assuming 90%. Reading config.json to Size a Model You Have Never Run and Weight Formats: FP8 Blocks, MXFP4 and AWQ cover those derivations.
Assume an additional 30 GB per rank for indexer/auxiliary state, workspaces and other allocations, and a 260 GB usable planning budget per rank. Ideal weight balance gives:
| Candidate | Weights/rank | Cache/rank | Other/rank | Total/rank | Result |
|---|---|---|---|---|---|
| 4 ranks | 203.438 GB | 29.528 GB | 30 GB | 262.965 GB | Exceeds budget |
| 8 ranks | 101.719 GB | 29.528 GB | 30 GB | 161.247 GB | Candidate for testing |
Even four ranks have 1,040 GB aggregate budget, more than 813.75 + 29.528 + 4×30 = 963.278 GB. That aggregate shortcut counts the cache once. The assumed layout holds four copies, so the physical total is 1,051.862 GB. Verify the actual cache placement before recommending the eight-rank candidate as well.
GB per rank, cache replication included
262.965 GB, over budget
161.247 GB, fits
260 GB
The proposed answer to the manager is: “Evaluate a supported eight-rank layout for 32 requests at 8K retained positions. The 30 GB runtime allowance and cache placement are unverified. First compare peak allocations by rank, then run task evaluation and a latency/load sweep.” This states a useful next step without calling an estimate a benchmark.
What interviewers probe next
- “What if quantization_config is absent?” Inspect tensor dtypes and the loader. A default dtype field is not proof that every stored tensor has that dtype.
- “Does non-native four-bit execution fail triage?” No. A fused conversion kernel may still benefit from lower memory traffic. Whole-tensor expansion can erase the memory saving; inspect the path and benchmark it.
- “Can you infer active parameters from token throughput?” No. Routing, reuse, cache traffic, kernel efficiency and communication prevent that inversion. Use tensor structure for parameter counts and a sweep for throughput.
- “What makes you change the plan?” Unsupported placement, a larger measured peak, or failed quality/latency requirements. A smaller model or a different supported layout may be better than adding GPUs.
Common mistakes
- Multiplying unrelated weight and cache errors together and calling the product a GPU-count error; those memory terms add.
- Counting cache once when the selected layout replicates it.
- Treating native low-bit arithmetic as necessary for any speedup.
- Assuming head/expert divisibility alone proves an engine supports a distributed layout.
- Promising a date from a repository card without testing initialization and the task contract.
Key takeaways
- Read architecture, artifact, runtime support and workload as separate inputs.
- The synthetic four-rank estimate misses the budget despite an apparent aggregate fit; replication changes the result.
- Give a candidate and a disqualifying test, then use measurements to finish the capacity plan.