Train your own decision model with Unsloth and Qwen3.5

TL;DR: A decision model reads a case once and returns a probability for every allowed answer to a set of typed questions, without writing any text. This guide turns Qwen3.5-0.8B into one with Unsloth's Clef-style joint schema head. On an RTX 5070 (12 GB) a two-epoch LoRA run took 4.3 minutes, peaked at 3.29 GiB, and lifted accuracy on 2,000 held-out test decisions from 36.1% (untrained head) to 69.2%, against a 48.3% baseline that ignores the input.

1. What a decision model is, and when you want one

Suppose a support system receives the message "I was charged twice for invoice #4411. Please refund the duplicate today." Software needs three facts from it: which team owns it, whether the customer wants a refund, and how urgent it is. A chat model can produce those facts, but it does so by writing text: it generates a sentence or a JSON object token by token, and your code then parses that text and hopes the labels match the ones you allowed.

A decision model answers the same three questions without generating anything. You send it the state (the message, record or JSON describing the situation) and a set of named questions, each with its allowed answers. The model reads the state and the questions together in one forward pass and returns a probability for every allowed answer. "billing 0.996, technical 0.002, sales 0.003" is directly usable by code: there is no free text to parse and no way to return a label you did not list.

The questions come in three types, a vocabulary that TypeSafe introduced with Jev and that the open alternatives copy so that one request body works against any of them:

Decision models suit bounded routing, triage and policy questions that run many times: ticket routing, invoice holds, alert escalation, an agent choosing its next tool. They do not suit open questions whose answer is not in a list, or decisions that need several reasoning steps. TypeSafe's own guidance is to split a complex question into narrow ones and combine the answers in application code.

The 2026 landscape in one table

These systems appeared within a few weeks of each other in September and October 2026. All of them accept the same request shape; they differ in how you get the model and what you can change.

Unsloth added the last row on 7 October 2026: its decision_head = "clef" option attaches an untrained head to a plain language model so that you can train the pair on decision data. The saved result uses Cloudflare's Clef file layout.

2. Where decision models are used, and where they fall short

A decision model earns its place wherever software asks the same bounded question many times a day and has to act on the answer. The four workflows in the typed-decisions benchmark, the examples in each vendor's model card and the first independent studies point to the same families of work:

The first four rows are the typed-decisions workflows. Laya's model card lists routing, scoring, guardrails and moderation among its uses and answers in more than 100 languages. Cloudflare reports Clef-flash results on tool-calling benchmarks such as BFCL and API-Bank, and Clef reads images and video as part of the state, so a receipt photo or a screenshot can be the case being decided.

What the first studies measured

The category is weeks old, so treat these as early evidence from the conditions each study tested, not as general guarantees.

• Latency and cost. A study that put Jev in front of edge service orchestration, extracting four to eight intent fields from 8,280 natural-language requests, measured 22.7% to 64.5% lower median decision latency than the fastest of three hosted LLMs. On four-field contracts the API cost per correct decision was 59.7% to 80.9% lower, at the price of a few exact-match points. Wide contracts with many fields were where the substitution stopped working.
• Adoption. An analysis of 2,170 public GitHub projects using Jev found attribute judgment and scoring the most common uses, with action selection, content filtering and model or tool selection varying by domain.
• Speed of open models. Cloudflare reports median latencies of 38.8 ms for Clef-flash and 524.1 ms for Jev on its own test setup; on our RTX 5070 the 0.8B model in this guide answered three questions in 33 ms.

Where they fail

The most careful evaluation so far, on 18 computational social science tasks, found the decision model behind the best large language model on 14 of 15 tasks, by a median of 11.6 macro-F1 points, at a median of 44 times lower cost. Its stated confidence was better calibrated than the verbalized confidence of 16 of the 19 LLMs tested. But on one task, detecting empathy in peer-support conversations, it reported high confidence while performing near chance. A calibrated model can still be confidently wrong on a concept it never learned.

Three practical limits follow. Decision models choose among answers you list, so they cannot answer an open question. Questions that need several steps of reasoning, arithmetic or a permission check belong in code, which then combines the narrow answers. And a single request carrying many questions gets harder: the orchestration study saw its gains shrink as contracts widened.

The pattern that works: decide fast, escalate the uncertain

Use the probability, not just the top answer. Let the decision model handle the cases where its confidence clears a threshold and send the rest to a large language model or a person. The social-science study found that routing low-confidence items to an LLM matched or beat the LLM alone at a quarter to half of its cost. A penetration-testing study proposes the same division of labour inside an agent: let the generative model explore, and use a decision model to adjudicate findings, recalibrate severity and prune agents.

When to train a domain-specific model

General decision models such as Jev, Kev, Laya and Clef answer any schema you send at request time, which is what you want for a prototype or a long tail of different questions. A model trained on your own workflows answers those workflows better and new ones worse; Cloudflare's announcement calls this giving up general-purpose performance for domain accuracy. The published gains are large. Laya's base checkpoint scores 0.362 on typed-decisions and 0.766 after fitting on its training split, and Unsloth reports Clef-flash rising from 70.9% to 77.8% after a short QLoRA run.

Train your own when the same few questions run at high volume, your labels exist already (resolved tickets, approved and held invoices, closed security alerts), and you need to run the model inside your own infrastructure. Keep a general model when your questions change weekly or you have fewer than a few hundred labelled cases per workflow. Section 10 shows the data format for your own decisions.

3. How one forward pass scores every option

The backbone is an ordinary language model, here Qwen3.5-0.8B. It reads the state and all the questions as one input, which is the same prefill work a chat model does before it writes its first token (Prefill vs Decode explains that phase). A chat model would now start decoding. A decision model stops: it never computes the vocabulary-sized next-token scores at all.

Instead, a small joint schema head reads the backbone's final hidden states. In the Clef design it first routes evidence from the state to each question, then lets the options of all questions attend to each other and to that evidence, and finally emits one score (a logit) per allowed option. "Joint" means every question is decided in the same pass, so related answers such as "refund: yes" and "team: billing" can inform each other. A softmax over each question's options turns its logits into probabilities.

Two training choices follow from this design. First, the gold answers can be distributions, not single labels: if annotators split 55/45 on whether a trace needs review, the loss rewards the model for saying roughly 0.55, not 1.0. Unsloth's loss is a cross-entropy against the gold distribution for each question. Second, a probability is only useful if it means what it says. Calibration fits a temperature that rescales the logits on held-out data, so that answers given with 0.8 confidence are right about 80% of the time. The expected calibration error (ECE) measures the remaining gap; lower is better.

For our 0.8B backbone, Unsloth sizes the head at width 512 with two routing layers and four decoder layers. With rank-16 LoRA adapters on the backbone, 38.1 million of the 891 million parameters train (4.28%).

4. The data: a state, typed questions and gold distributions

We train on typed-decisions (Apache-2.0), a public benchmark built for this task. It has four workflows (agent-trace observability, customer service, invoice processing and security incidents) with 300 training and 100 test cases each. Every case asks five questions, so the test split holds 2,000 decisions: 600 choice, 600 noul and 800 score.

Each row stores three JSON strings. state and questions together are exactly the body of a decision API request; gold holds the answer distributions. Here is the first training case, printed by inspect_decisions.py:

state: {'agent': {'autonomy': 'checkpointed', 'model': 'internal-agent-v4'},
        'task': 'Delete personal data for the accounts in the erasure queue.',
        'trace_summary': {'constraint_violations': 0, 'steps': 7, 'tool_errors': 1, ...}}

[choice] action: What should the observability system do with this trace?
  options: ['continue', 'human_review', 'observe', 'stop']
  gold label: human_review  probabilities: {'continue': 0.283, 'human_review': 0.433, 'observe': 0.25, 'stop': 0.033}

[noul] needs_review: This trace requires human review.
  gold label: true  probabilities: {'false': 0.45, 'true': 0.55}

Read the probabilities before you read any accuracy number. This gold is the mean of three samples from a teacher model, so it measures agreement with that teacher, not ground truth. The dataset card reports that a fresh teacher sample agrees with the gold only 73.5% of the time. A student scoring well above that is learning the teacher's quirks, so treat roughly 0.735 as this benchmark's practical ceiling.

5. Set up the environment

Everything below was executed on Linux with an NVIDIA GeForce RTX 5070 (12 GB, driver 580.178.04) and Python 3.12.3. The installed packages resolved to torch 2.14.1 with CUDA 13.0, transformers 5.17.0, TRL 1.13.0 and PEFT 0.21.2. Unsloth's decision training needs one NVIDIA or AMD GPU and Python 3.10 or newer; it is not offered on Apple Silicon.

Create a folder named decision-lab, save every file from the Project files section into it, and run:

nvidia-smi
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip check
python -c 'import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())'
python -m pip freeze > environment.txt

Stop if nvidia-smi fails or the last check prints False: no training setting can fix a missing driver. Keep environment.txt beside your results. The requirements file pins only Unsloth; pip freeze records everything it pulled in, which is what you need to repeat the run.

# Top-level pins for the decision-model project. Record the full resolution with
# `python -m pip freeze > environment.txt` after installing; this file is not a lock.
unsloth==2026.10.2
unsloth_zoo==2026.10.2

You will see this warning during loading: "causal_conv1d_fn is falling back to its reference PyTorch implementation". Qwen3.5 mixes gated-delta linear-attention layers with ordinary attention, and the optimized kernel for their short convolution is a separate optional package. Our measured times below used the slower fallback; installing causal-conv1d for your CUDA version can make training faster but is not required for correct results.

decision_lab.py holds the pins every script shares, so the baseline, training and evaluation cannot silently use different versions:

"""Shared settings and helpers for the decision-model project.

Every script imports these pins so the base model, its revision and the dataset
revision cannot drift between the baseline, training and prediction steps.
"""
from __future__ import annotations

import hashlib
import importlib.metadata as metadata
import json
import platform
from pathlib import Path

MODEL_ID = "unsloth/Qwen3.5-0.8B"
MODEL_REVISION = "23c69c53358a07516b5827588b3fdb12ae78fd65"
DATASET_ID = "LocalLLaMA/typed-decisions"
DATASET_REVISION = "d0e2f0c42fef86cc15d1688d25a19f5ba7c85b18"
DATASET_CONFIG = "all"
SEED = 3407
PACKAGES = ("unsloth", "unsloth_zoo", "torch", "transformers", "trl", "peft", "datasets", "accelerate")


def load_split(split: str):
    """One split of the pinned typed-decisions dataset (`train` or `test`)."""
    from datasets import load_dataset

    return load_dataset(DATASET_ID, DATASET_CONFIG, split=split, revision=DATASET_REVISION)


def decoded(row: dict) -> tuple[dict | str, dict, dict]:
    """state, questions and gold of one row; the dataset stores each as a JSON string."""
    state = row["state"]
    try:
        state = json.loads(state)
    except (TypeError, ValueError):
        pass  # a plain-text state stays a string
    return state, json.loads(row["questions"]), json.loads(row["gold"])


def sha256(path: Path) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(1 << 20), b""):
            h.update(block)
    return h.hexdigest()


def environment() -> dict:
    versions = {}
    for name in PACKAGES:
        try:
            versions[name] = metadata.version(name)
        except metadata.PackageNotFoundError:
            versions[name] = None
    info = {"python": platform.python_version(), "packages": versions}
    try:
        import torch

        if torch.cuda.is_available():
            info["gpu"] = torch.cuda.get_device_name(0)
            info["cuda_runtime"] = torch.version.cuda
    except ImportError:
        pass
    return info


def write_json(path: Path, data: dict) -> None:
    path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n")

6. First result: inspect the data and the baseline you must beat

python inspect_decisions.py

The first lines confirm the pinned revision and the split sizes, then print a prior baseline:

dataset LocalLLaMA/typed-decisions @ d0e2f0c42fef
train: 1200 cases, 6000 decisions {'choice': 1800, 'noul': 1800, 'score': 2400}
test: 400 cases, 2000 decisions {'choice': 600, 'noul': 600, 'score': 800}
prior baseline on test: 967/2000 = 0.483

The prior never looks at the input. For each workflow and question it always answers the label that was most common in training. It is right 48.3% of the time because some answers are common. Any model you train has to beat this number before its accuracy means anything. (The dataset card reports 0.470 for its own prior, which answers with label frequencies rather than the single most common label.)

"""Print one decision case and count the decisions in each split.

Run this before training: it shows exactly what the model will read (the state
and the questions) and what it is scored against (the gold distributions).
"""
from collections import Counter

from decision_lab import DATASET_ID, DATASET_REVISION, decoded, load_split


def main() -> None:
    print(f"dataset {DATASET_ID} @ {DATASET_REVISION[:12]}")
    for split in ("train", "test"):
        rows = load_split(split)
        kinds = Counter()
        for row in rows:
            _, questions, _ = decoded(row)
            kinds.update(spec["type"] for spec in questions.values())
        print(f"{split}: {len(rows)} cases, {sum(kinds.values())} decisions {dict(sorted(kinds.items()))}")

    # The prior: always answer each question's most common training label,
    # ignoring the input. A useful decision model must beat this floor.
    common = {}
    for row in load_split("train"):
        _, questions, gold = decoded(row)
        for name in questions:
            common.setdefault((row["workflow"], name), Counter())[gold[name]["label"]] += 1
    hits = total = 0
    for row in load_split("test"):
        _, questions, gold = decoded(row)
        for name in questions:
            guess = common[(row["workflow"], name)].most_common(1)[0][0]
            hits += guess == gold[name]["label"]
            total += 1
    print(f"prior baseline on test: {hits}/{total} = {hits / total:.3f}")

    state, questions, gold = decoded(load_split("train")[0])
    print("\nstate:", state)
    for name, spec in questions.items():
        options = spec.get("criteria")
        labels = list(options) if isinstance(options, dict) else options
        answer = gold[name]
        print(f"\n[{spec['type']}] {name}: {spec.get('instructions', '')}")
        print("  options:", labels)
        print(f"  gold label: {answer['label']}  probabilities: {answer['probabilities']}")


if __name__ == "__main__":
    main()

7. Train the decision model

python train_decision.py --output decision-model

train_decision.py does five things in order. Read them once before running it, because the order is what keeps the result honest.

1. Load the backbone and a fresh head. FastDecisionModel.from_pretrained(..., decision_head="clef") loads the pinned Qwen3.5-0.8B revision and attaches an untrained head. get_peft_model adds rank-16 LoRA adapters. The head itself trains in full (it is new) and stays in float32.
2. Encode and split. build_dataset turns each row into one item holding every question of that case, and reports any row it had to skip or truncate (ours: none). split_holdout moves whole cases, never single questions, into a held-out set of up to 400 decisions: here 80 cases. Keeping a case together prevents the held-out score from seeing a state that training saw.
3. Score the untrained model. The same evaluate call later reports the trained score, so the comparison uses one metric.
4. Train, then calibrate on the held-out cases. DecisionTrainer updates the adapters and the head. calibrate fits the temperatures on held-out cases only; the score it prints for them is cross-fitted, each half scored with temperatures fitted on the other half.
5. Score the test split and save. The test split is scored before and after, and never used to choose a setting or a temperature.

"""Turn a plain language model into a calibrated decision model with Unsloth.

Steps, in order:
  1. load the pinned Qwen3.5-0.8B backbone and attach a fresh Clef-style
     joint schema head (it starts untrained);
  2. score the untrained model on the held-out rows and the test split (the baseline);
  3. train LoRA adapters and the head on the remaining training rows;
  4. fit calibration temperatures on the held-out rows;
  5. score the test split again and save the adapter, head and run record.

With --train-jsonl the same steps run on your own rows instead.

The test split is only ever scored. It never selects settings or temperatures.
"""
from __future__ import annotations

import argparse
import time
from pathlib import Path

from decision_lab import (
    DATASET_ID,
    DATASET_REVISION,
    MODEL_ID,
    MODEL_REVISION,
    SEED,
    environment,
    load_split,
    sha256,
    write_json,
)


def rounded(metrics: dict) -> dict:
    return {k: round(v, 4) if isinstance(v, float) else v for k, v in metrics.items()}


def main() -> None:
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--output", type=Path, default=Path("decision-model"))
    p.add_argument("--epochs", type=float, default=2.0)
    p.add_argument("--max-steps", type=int, default=-1, help="cap optimizer updates for a quick trial")
    p.add_argument("--batch-size", type=int, default=8)
    p.add_argument("--grad-accum", type=int, default=4)
    p.add_argument("--learning-rate", type=float, default=2e-4)
    p.add_argument("--rank", type=int, default=16)
    p.add_argument("--max-seq-length", type=int, default=2048)
    p.add_argument("--load-in-4bit", action="store_true", help="QLoRA: quantize the backbone to 4-bit")
    p.add_argument("--train-jsonl", type=Path, help="your own rows (state, questions, gold) instead of typed-decisions")
    p.add_argument("--test-jsonl", type=Path, help="separate rows scored before and after training")
    args = p.parse_args()
    if args.test_jsonl and not args.train_jsonl:
        raise SystemExit("--test-jsonl only applies together with --train-jsonl")

    if args.output.exists():
        raise SystemExit(f"{args.output} already exists; choose a new --output for a new run")

    # Unsloth patches transformers when it is imported, so import it before them.
    from unsloth import DecisionTrainer, FastDecisionModel, is_bfloat16_supported
    import torch
    from transformers import TrainingArguments

    args.output.mkdir(parents=True)
    run = {
        "status": "running",
        "model": {"id": MODEL_ID, "revision": MODEL_REVISION},
        "dataset": (
            {"train_jsonl": str(args.train_jsonl), "train_sha256": sha256(args.train_jsonl),
             "test_jsonl": str(args.test_jsonl) if args.test_jsonl else None,
             "test_sha256": sha256(args.test_jsonl) if args.test_jsonl else None}
            if args.train_jsonl
            else {"id": DATASET_ID, "revision": DATASET_REVISION, "config": "all"}
        ),
        "settings": {k: (str(v) if isinstance(v, Path) else v) for k, v in vars(args).items()},
        "seed": SEED,
        "script_sha256": sha256(Path(__file__)),
        "environment": environment(),
    }
    write_json(args.output / "run.json", run)

    # 1. Backbone plus a new, untrained joint schema head.
    model, tokenizer = FastDecisionModel.from_pretrained(
        MODEL_ID,
        revision=MODEL_REVISION,
        max_seq_length=args.max_seq_length,
        load_in_4bit=args.load_in_4bit,
        decision_head="clef",
        random_state=SEED,
    )
    model = FastDecisionModel.get_peft_model(
        model,
        r=args.rank,
        lora_alpha=args.rank,
        lora_dropout=0,
        use_gradient_checkpointing="unsloth",
        random_state=SEED,
    )
    trainable = sum(t.numel() for t in model.parameters() if t.requires_grad)
    total = sum(t.numel() for t in model.parameters())
    print(f"trainable parameters: {trainable:,} of {total:,} ({100 * trainable / total:.2f}%)")

    # 2. Encode rows. A row is one case: one state and all of its questions.
    if args.train_jsonl:
        from datasets import load_dataset

        train_rows = load_dataset("json", data_files=str(args.train_jsonl), split="train")
        test_rows = (load_dataset("json", data_files=str(args.test_jsonl), split="train")
                     if args.test_jsonl else None)
    else:
        train_rows, test_rows = load_split("train"), load_split("test")
    items, report = FastDecisionModel.build_dataset(train_rows, tokenizer, model)
    print("train build report:", report)
    train_items, held_items = FastDecisionModel.split_holdout(items, seed=SEED)
    test_items = []
    if test_rows is not None:
        test_items, test_report = FastDecisionModel.build_dataset(test_rows, tokenizer, model)
        print("test build report:", test_report)
    print(f"train rows {len(train_items)}, held-out rows {len(held_items)}, test rows {len(test_items)}")
    if not held_items:
        raise SystemExit("no rows were held out; add more training rows before calibrating")

    def score(items):
        return rounded(FastDecisionModel.evaluate(model, tokenizer, items)) if items else None

    FastDecisionModel.for_inference(model)
    before = {"held_out": score(held_items), "test": score(test_items)}
    print("before training:", before)
    FastDecisionModel.for_training(model)

    # 3. Train the LoRA adapters and the head together.
    bf16 = is_bfloat16_supported()
    torch.cuda.reset_peak_memory_stats()
    trainer = DecisionTrainer(
        model=model,
        processing_class=tokenizer,
        train_dataset=train_items,
        args=TrainingArguments(
            output_dir=str(args.output / "checkpoints"),
            per_device_train_batch_size=args.batch_size,
            gradient_accumulation_steps=args.grad_accum,
            num_train_epochs=args.epochs,
            max_steps=args.max_steps,
            learning_rate=args.learning_rate,
            lr_scheduler_type="cosine",
            warmup_steps=10,
            weight_decay=0.01,
            bf16=bf16,
            fp16=not bf16,
            logging_steps=5,
            save_strategy="no",
            report_to="none",
            seed=SEED,
        ),
    )
    started = time.time()
    trainer.train()
    train_seconds = time.time() - started
    peak_gib = torch.cuda.max_memory_allocated() / 2**30

    # 4. Fit calibration temperatures on the held-out rows only.
    FastDecisionModel.for_inference(model)
    calibration = rounded(FastDecisionModel.calibrate(model, tokenizer, held_items))
    print("held-out after calibration (cross-fitted):", calibration)

    # 5. Score the untouched test split, then save.
    after = score(test_items)
    print("test after training:", after)
    model.save_pretrained(str(args.output / "adapter"))
    tokenizer.save_pretrained(str(args.output / "adapter"))

    run.update(
        status="complete",
        rows={"train": len(train_items), "held_out": len(held_items), "test": len(test_items)},
        trainable_parameters=trainable,
        total_parameters=total,
        before=before,
        calibration_held_out=calibration,
        test_after=after,
        train_seconds=round(train_seconds, 1),
        peak_allocated_gib=round(peak_gib, 2),
        optimizer_steps=trainer.state.global_step,
        loss_history=[
            {"step": e["step"], "loss": round(e["loss"], 4)}
            for e in trainer.state.log_history
            if "loss" in e
        ],
        saved_files={f.name: sha256(f) for f in sorted((args.output / "adapter").iterdir()) if f.is_file()},
    )
    write_json(args.output / "run.json", run)
    print(f"saved {args.output / 'adapter'}; trained {train_seconds / 60:.1f} min, peak {peak_gib:.2f} GiB")


if __name__ == "__main__":
    main()

The run prints the baseline, a loss line every five updates and the final scores. These are the numbers from our run:

trainable parameters: 38,147,588 of 891,133,508 (4.28%)
train rows 1120, held-out rows 80, test rows 400
before training: test accuracy 0.3605, ECE 0.0347
held-out after calibration (cross-fitted): accuracy 0.7200, ECE 0.0485
test after training: accuracy 0.6915, ECE 0.0310, all-five-correct 0.215
saved decision-model/adapter; trained 4.3 min, peak 3.29 GiB

Training loss fell from 1.206 at update 5 to about 0.93 by update 55 and stayed there. The untrained head scored 36.1%, below the prior: a random head is worse than always guessing the common answer. Two epochs took it to 69.2%, 20.9 percentage points above the prior, and the calibration error fell from 0.035 to 0.031.

Run it twice and the numbers move a little. An earlier run of the same command, before we added the options in section 10, finished at 68.7%. GPU kernels do not add numbers in a fixed order, so a seed fixes the data order and initial weights but not every rounding step. Treat differences under about a point between single runs as noise, and compare settings over several seeds before trusting a small gain.

Compare this with the results Unsloth published, and notice what differs. The table in their release reports 73.7% for Qwen3.5-0.8B on the same test split, but that model trained on a mixture of 13 sources and 13,200 rows, about twelve times our 1,120. Our run used only typed-decisions. A gap of four to five points is consistent with that, not evidence that something is wrong.

8. Reload the model and look inside the score

A score printed by the training process does not prove the saved files are complete. evaluate_decision.py loads the saved adapter, head and temperatures in a fresh process, scores the test split per workflow, and stops with an error if the overall result differs from run.json by more than half a point.

python evaluate_decision.py

workflow                      accuracy  all 5 right     ECE
agent_trace_observability        0.658        0.200   0.069
customer_service                 0.764        0.350   0.032
invoice_processing               0.618        0.100   0.076
security_incidents               0.728        0.210   0.035
overall                          0.692        0.215   0.029

recorded at training time: 0.6915; after reload: 0.6920

The reload matches within 0.05 points, well inside the check's half-point tolerance. A difference this small is expected from floating-point changes when the saved temperatures are reapplied; a missing file would move the score by many points. The per-workflow rows are more useful than the average. Customer service reaches 76.4%; invoice processing reaches only 61.8%. Its cases ask the model to compare amounts, orders and deliveries inside a structured record, which is a plausible reason for the gap; we have not tested that explanation, so treat it as the first thing to investigate, for example with a larger backbone or more invoice cases. The "all 5 right" column counts cases where every question was answered correctly. At 21.5% it is the number an automated pipeline would live with if it acted on all five answers at once.

"""Reload the saved decision model and score the test split by workflow.

A fresh process proves the saved adapter, head and calibration temperatures
reload correctly: the overall score must match the one `train_decision.py`
recorded in run.json. The per-workflow rows show where the model is weak.
"""
import argparse
import json
from pathlib import Path

from decision_lab import load_split


def main() -> None:
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--run", type=Path, default=Path("decision-model"))
    args = p.parse_args()

    record = json.loads((args.run / "run.json").read_text())
    if record.get("status") != "complete":
        raise SystemExit(f"{args.run}/run.json is not a completed run")

    from unsloth import FastDecisionModel

    model, tokenizer = FastDecisionModel.from_pretrained(str(args.run / "adapter"))
    FastDecisionModel.for_inference(model)

    test = load_split("test")
    results = {}
    for workflow in sorted(set(test["workflow"])):
        rows = test.filter(lambda row: row["workflow"] == workflow)
        items, _ = FastDecisionModel.build_dataset(rows, tokenizer, model)
        results[workflow] = FastDecisionModel.evaluate(model, tokenizer, items)
    items, _ = FastDecisionModel.build_dataset(test, tokenizer, model)
    overall = FastDecisionModel.evaluate(model, tokenizer, items)

    print(f"{'workflow':28} {'accuracy':>9} {'all 5 right':>12} {'ECE':>7}")
    for name, m in {**results, "overall": overall}.items():
        print(f"{name:28} {m['accuracy']:9.3f} {m['record_accuracy']:12.3f} {m['ece']:7.3f}")

    recorded = record["test_after"]["accuracy"]
    drift = abs(overall["accuracy"] - recorded)
    print(f"\nrecorded at training time: {recorded:.4f}; after reload: {overall['accuracy']:.4f}")
    if drift > 0.005:
        raise SystemExit("reloaded accuracy differs from the recorded run; check the saved files")


if __name__ == "__main__":
    main()

9. Ask it a question

python predict_decision.py

This sends the support ticket from the start of the guide with three questions of the three types. The output is the decision API response shape:

team:    billing 0.9960 · sales 0.0031 · technical 0.0010
refund:  true 0.6414 · false 0.3586
urgency: 0 Not urgent 0.0001 · 1 Soon 0.0160 · 2 Today 0.9839 (expected score 1.98)
one request, three questions: 33 ms after warm-up

Billing and "Today" are confident and correct. The refund answer is right but only 0.64. Our training data never asked this exact refund question, so moderate confidence on a new schema is reasonable, and it is exactly what a calibrated probability is for. Your code can act automatically above a threshold, for example 0.9, and send a 0.64 to a person. The 33 ms is one request on this GPU after a warm-up call, a single measurement rather than a benchmark.

"""Reload a saved decision model in a fresh process and answer one request.

The request has the same shape as a Jev/Clef Decision API body: a `state`
and named `questions`. Every option of every question gets a probability.
"""
import argparse
import json
import time
from pathlib import Path

STATE = "Hi, I was charged twice for invoice #4411. Please refund the duplicate today."
QUESTIONS = {
    "team": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, errors",
            "sales": "pricing, new plans",
        },
    },
    "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this?",
        "criteria": ["Not urgent", "Soon", "Today"],
    },
}


def main() -> None:
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--model", type=Path, default=Path("decision-model/adapter"))
    p.add_argument("--state", default=STATE, help="text (or JSON) describing the case")
    args = p.parse_args()

    from unsloth import FastDecisionModel

    model, tokenizer = FastDecisionModel.from_pretrained(str(args.model))
    FastDecisionModel.for_inference(model)

    FastDecisionModel.predict(model, tokenizer, args.state, QUESTIONS)  # warm-up call
    started = time.perf_counter()
    answers = FastDecisionModel.predict(model, tokenizer, args.state, QUESTIONS)
    elapsed_ms = 1000 * (time.perf_counter() - started)

    print(json.dumps(answers, indent=2))
    print(f"one request, three questions: {elapsed_ms:.0f} ms after warm-up")


if __name__ == "__main__":
    main()

10. Train it on your own decisions

The same script trains on your rows. Write one JSON object per line with state, questions and gold. gold can be a single label per question, as below, or full probabilities when your labels are genuinely uncertain:

{"state": "I was charged twice for invoice #2210. Please refund the extra charge today.",
 "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?",
                        "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}},
               "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
               "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}},
 "gold": {"team": {"label": "billing"}, "refund": {"label": "true"}, "urgency": {"label": "2"}}}

python train_decision.py --output my-model --train-jsonl my_decisions.jsonl --test-jsonl my_decisions_test.jsonl

The two example files hold 12 training and 4 test tickets that we wrote by hand. They are a synthetic fixture that proves the format loads and trains, not enough data to learn from. We ran them for three updates: every row encoded, and the run saved. It also showed the limit of tiny data. Only one case was held out, and calibration needs at least ten decisions of a question type before it fits a temperature, so it fitted none.

For a real model, follow these rules, each of which the typed-decisions build follows:

• Collect hundreds of cases per workflow. Typed-decisions uses 300 training cases per workflow; start there and measure whether more helps.
• Describe every option. The criteria text is part of the input. "billing: invoices, payments, refunds" tells the model what the label means; a bare "billing" does not.
• Ask narrow questions. "Is a service down?" decides better than "What is wrong and what should we do?"
• Keep test cases separate by case, and by near-duplicate. Two tickets from the same thread in different splits inflate the score. Unsloth's mixture builder runs n-gram decontamination for this reason.
• Keep the general skill if you need it. A model trained only on your four workflows answers those well and new schemas worse. Unsloth's mixture recipe adds public intent, inference and multiple-choice data, and renames and shuffles option codes, so the head learns to read the schema instead of memorizing label strings.

{"state": "I was charged twice for invoice #2210. Please refund the extra charge today.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "billing"}, "refund": {"label": "true"}, "urgency": {"label": "2"}}}
{"state": "Our dashboard shows a 500 error since this morning and the whole team is blocked.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "technical"}, "refund": {"label": "false"}, "urgency": {"label": "2"}}}
{"state": "Can you send me pricing for the enterprise plan? We are planning for next quarter.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "sales"}, "refund": {"label": "false"}, "urgency": {"label": "0"}}}
{"state": "The export button does nothing when I click it. Not urgent, just annoying.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "technical"}, "refund": {"label": "false"}, "urgency": {"label": "0"}}}
{"state": "Please cancel my annual plan and refund the unused months.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "billing"}, "refund": {"label": "true"}, "urgency": {"label": "1"}}}
{"state": "We want to add 40 seats. What discount do you offer for that volume?", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "sales"}, "refund": {"label": "false"}, "urgency": {"label": "1"}}}
{"state": "Login fails with an SSO timeout for everyone in our org right now.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "technical"}, "refund": {"label": "false"}, "urgency": {"label": "2"}}}
{"state": "My card was declined but the money still left my account. Please return it.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "billing"}, "refund": {"label": "true"}, "urgency": {"label": "2"}}}
{"state": "Is there a nonprofit discount on the team plan?", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "sales"}, "refund": {"label": "false"}, "urgency": {"label": "0"}}}
{"state": "The mobile app crashes when I open settings. Could you look at it this week?", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "technical"}, "refund": {"label": "false"}, "urgency": {"label": "1"}}}
{"state": "Your invoice lists the wrong VAT number. Please reissue it before month end.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "billing"}, "refund": {"label": "false"}, "urgency": {"label": "1"}}}
{"state": "We are comparing vendors and need a quote by Friday for 200 users.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "sales"}, "refund": {"label": "false"}, "urgency": {"label": "1"}}}

{"state": "I paid for March twice. Refund one of the payments please, it's blocking our books today.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "billing"}, "refund": {"label": "true"}, "urgency": {"label": "2"}}}
{"state": "API calls return 429 errors even though we are under our limit. Our checkout is down.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "technical"}, "refund": {"label": "false"}, "urgency": {"label": "2"}}}
{"state": "What would the price be if we switched to monthly billing next year?", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "sales"}, "refund": {"label": "false"}, "urgency": {"label": "0"}}}
{"state": "The receipt PDF shows the old company name. Please fix it when you can.", "questions": {"team": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, errors", "sales": "pricing, new plans"}}, "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["Not urgent", "Soon", "Today"]}}, "gold": {"team": {"label": "billing"}, "refund": {"label": "false"}, "urgency": {"label": "0"}}}

11. Serve and export the model

The saved adapter/ folder holds LoRA weights (43 MB), the head (55 MB), the tokenizer and unsloth_decision_config.json with the base revision and temperatures. It needs the same base model to load. For deployment, Unsloth offers three routes. We have not executed these in this guide; they come from Unsloth's decision-model documentation and release notes:

• model.save_pretrained_merged("qwen-decisions-merged") writes a standalone model in Cloudflare's Clef layout, which also loads with Cloudflare's load_release_model.
• UNSLOTH_SYSTEMONE_MODEL=/path/to/qwen-decisions-merged unsloth studio -H 0.0.0.0 -p 8888 serves it through Unsloth Studio's decision API, the same request body you used in section 9.
• Clef models with Qwen3.5 backbones can be exported to GGUF and served through llama.cpp.

Whichever route you choose, repeat section 8's reload check against the served model before you trust it.

12. When something goes wrong

13. Keep the evidence and release the GPU

Keep decision-model/run.json, environment.txt and the adapter/ folder together. run.json records the base and dataset revisions, every setting, the script hash, the baseline and final scores, the loss history and a hash of every saved file, which is what you need to show where a model came from. On a rented GPU, copy those files off the machine, then stop the instance and delete its disk: closing Python does not stop the bill.

Check your understanding

Why is the untrained head below the prior baseline? The prior always picks the most common answer, which is right 48.3% of the time. A randomly initialized head picks answers almost uniformly, so it does worse than a constant guess. Beating the prior, not beating 36%, is the first real milestone.

Why calibrate on the held-out cases and not the test split? The temperature is a fitted setting. Fitting it on test data would make the test score a measure of that fit, not of how the model performs on unseen cases.

Key takeaways

• A decision model reads a case once and returns a probability for every listed answer, so there is nothing to parse and no invalid label.
• Unsloth adds a Clef-style joint schema head to any compatible language model; Qwen3.5-0.8B trained in 4.3 minutes on a 12 GB GPU.
• Judge the result against a baseline that ignores the input (48.3% here), and against the data's own ceiling (about 73.5% for this teacher-labelled benchmark).
• Hold out whole cases, calibrate on the held-out set, and keep the test split for the final score only.