Prepare a JSONL dataset for small-model fine-tuning

TL;DR: Define one classification task, validate every JSONL record and keep related examples in the same data split. The included script produces training, validation and test files, counts each label and records file hashes. Run the synthetic fixture first, then replace it with reviewed data you have permission to use.

1. Define the behavior before collecting rows

The example task maps a short incident description to exactly one label: MEMORY, NETWORK, or SCHEDULER. An output such as “This looks like a memory problem” fails the chosen format, even though a person can infer its meaning. Choosing the output contract now gives you an objective check after training.

These labels are a teaching exercise. A real incident can have several causes, and a short sentence may not contain enough evidence to diagnose it. A production classifier would need an uncertainty or escalation policy, more representative examples and review by someone responsible for the operational decisions. Do not treat our synthetic labels as an incident-response system.

A JSONL file contains one JSON object per line. Keeping each example on one line makes it possible to stream large files and identify the specific record that fails validation. Hugging Face's dataset loading guide supports this representation directly.

2. Keep source identity with every example

A record starts in this shape:

{"group":"incident-017","messages":[{"role":"user","content":"Classify as MEMORY, NETWORK, or SCHEDULER: CUDA reports out of memory at startup"},{"role":"assistant","content":"MEMORY"}]}

group identifies the original incident or source, not a unique rewrite. If you produce five paraphrases of incident 017, all five retain incident-017. Otherwise the trainer may see one version while the evaluator tests another, making memorization look like generalization.

The script expects precisely two messages in user/assistant order. It does not accept multi-turn dialogue, tool calls, image inputs or several labels in one answer. Those are different schemas. Enforcing a narrow schema is useful because every surviving row has the same interpretation.

Before using real records, remove credentials and unnecessary personal data, check usage rights and have the labels reviewed. A schema validator cannot establish permission, detect all sensitive information or judge whether a label is technically correct. Keep those decisions in the source dataset's review notes.

3. Run the CPU fixture

Download prepare_data.py into an empty folder. Python 3.12 and its standard library are sufficient; there is no package installation or account requirement for this step.

mkdir slm-data
cd slm-data
# Save the downloaded prepare_data.py here before continuing.
python3.12 prepare_data.py --output prepared

Save the downloaded script in slm-data before running the final command. With no --input, it creates an original synthetic fixture: six source groups per label, with two formulations per group. No customer data or external dataset is fetched.

You get four files:

Each split includes every label. Because the fixture is small, reserving one group per label produces a 12/3/3 group split, rather than exactly 80/10/10. For larger per-label groups, the script reserves the integer part of 10% for each held-out split, with at least one group per label. Counts describe groups; uneven group sizes can produce different row proportions.

4. Follow the checks that protect the split

prepare() first validates and deduplicates records, then builds groups, a dictionary from a source ID to all its rows. It shuffles the list of group IDs within each label, assigns those IDs to splits, and only then expands each group back into rows. The order matters: shuffling individual rows first would discard the protection the group ID provides.

The checks run before the output directory is created. Empty messages, unexpected roles and unknown labels stop the run. Prompts are normalized for Unicode form, case and whitespace when checking exact duplicates. Duplicate prompts with the same group and answer are removed; a conflicting group or answer stops the run so you can resolve the source record deliberately.

This normalization does not find paraphrases. “The peer timed out” and “No response came back from the other worker” can still describe the same event. That is why the source group is required even when exact duplicate checks pass.

The script stratifies by label: it distributes each label's source groups across the three splits. This version requires all rows in one source group to carry one label. For a dataset with mixed-label groups, use a splitter designed for that constraint and check its resulting distributions. Do not split the group into individual rows merely to satisfy the current script.

The default output directory must not already exist. This protects a previously inspected split from being overwritten during an accidental rerun. Choose a new output name when creating a new dataset version.

5. Inspect what the trainer will receive

Output records use a conversational prompt/completion format:

{"prompt":[{"role":"user","content":"Classify as MEMORY, NETWORK, or SCHEDULER: CUDA reports out of memory at startup"}],"completion":[{"role":"assistant","content":"MEMORY"}]}

The source group stays in manifest.json. The trainer receives the prompt and desired completion, without a group identifier that might become an accidental prediction shortcut. The TRL dataset format reference describes this prompt/completion boundary.

Inspect several rows and the manifest:

python3.12 -m json.tool prepared/manifest.json
head -n 2 prepared/train.jsonl
wc -l prepared/train.jsonl prepared/valid.jsonl prepared/test.jsonl

The head and wc commands are Linux shell utilities. json.tool is appropriate for the single manifest object; a JSONL file contains several objects and should be read line by line.

A file hash identifies the exact bytes used. If you alter punctuation, reorder rows or change an answer, the hash changes. Preserve the split files and manifest with each model run so you can distinguish a model change from a data change. Evaluation and data infrastructure explains the wider release relationship; that deeper concept requires Premium.

6. Add your own examples and deliberately break one

Put reviewed source records in incidents.jsonl and run:

python3.12 prepare_data.py --input incidents.jsonl --output prepared-v2

Supply at least three independent groups for each of the three labels. This minimum only lets the splitter reserve every label in every split; it is far too small to establish deployment quality. The fixture's six-row test set is especially noisy: one additional error changes the fraction by about 16.7 percentage points.

Before trusting the validator, duplicate one prompt with a different assistant label and rerun into another unused directory. The run should fail with a duplicate/conflict error. Restore the original, change a role to system, and verify that the role-order check fails. A validator that has only been exercised on good data has not demonstrated its rejection behavior.

7. Freeze the evaluation boundary

Use validation to choose a learning rate, training length or prompt format. Open the final test results only after those decisions are fixed. Once you repeatedly change the system in response to test errors, that set is serving as validation; reserve new independent data for the next final check. The scikit-learn leakage guidance explains why information from held-out data must stay outside fitting decisions.

Group separation is one defense. Also ask whether the groups share a template, an upstream generated source, an author or a collection period that makes evaluation artificially easy. For time-dependent incidents, a chronological split may answer the deployment question better than this shuffled exercise. Change the method when the source relationship demands it, and record the method in the manifest.

Keep prepared/ and its manifest for the Unsloth fine-tuning project, available with a free account. The CPU training guide is another free next step if you want to inspect how targets become loss. No background process or cloud resource remains after this data script exits.

Check your understanding

You have 100 rewrites of one incident. How many independent source groups did you add? One. A larger row count does not create new incidents for evaluation.

If two differently worded examples came from the same support ticket, what must match? Their group values. Different file hashes and different wording cannot establish that their sources are independent. Inspect the manifest's group lists before moving prepared/ into the training project.

Complete project source

These are the same files offered under Project files. Run the downloaded files; the listings let you inspect each implementation.

prepare_data.py

"""Validate two-message classification examples and split entire source groups."""
import argparse
import hashlib
import json
import random
import unicodedata
from pathlib import Path

LABELS = {'MEMORY', 'NETWORK', 'SCHEDULER'}
EXAMPLES = {
    'MEMORY': ['GPU allocation failed during model loading', 'Cache allocation exceeds free device memory',
               'Longer prompts exhaust device memory', 'CUDA reports out of memory at startup',
               'Increasing the batch causes allocation failure', 'Another process holds most GPU memory'],
    'NETWORK': ['The worker cannot reach the peer address', 'Packets to the peer are dropped',
                'The connection to the peer times out', 'The network interface has no link',
                'DNS lookup for the worker fails', 'The firewall blocks the peer port'],
    'SCHEDULER': ['The job is pending because its GPU quota is exhausted', 'No node matches the requested GPU label',
                  'The queue is paused by its administrator', 'The job waits for a gang allocation',
                  'The requested GPU resources exceed the queue limit', 'The node is marked unschedulable'],
}


def normalize(text):
    return ' '.join(unicodedata.normalize('NFKC', text).casefold().split())


def prepare(rows, output):
    groups, seen = {}, {}
    for number, row in enumerate(rows, 1):
        group = row.get('group')
        messages = row.get('messages')
        if not isinstance(group, str) or not group.strip():
            raise ValueError(f'Row {number}: a nonempty source group is required')
        if not isinstance(messages, list) or len(messages) != 2:
            raise ValueError(f'Row {number}: expected exactly user and assistant messages')
        if any(not isinstance(m, dict) for m in messages):
            raise ValueError(f'Row {number}: messages must be objects')
        if [m.get('role') for m in messages] != ['user', 'assistant']:
            raise ValueError(f'Row {number}: incorrect role order')
        if any(not isinstance(m.get('content'), str) or not m['content'].strip() for m in messages):
            raise ValueError(f'Row {number}: empty or non-text content')
        prompt, answer = (m['content'].strip() for m in messages)
        if answer not in LABELS:
            raise ValueError(f'Row {number}: unknown label {answer}')
        key = normalize(prompt)
        if key in seen:
            previous_group, previous_answer = seen[key]
            if (previous_group, previous_answer) != (group, answer):
                raise ValueError(f'Row {number}: duplicate prompt has a different group or label')
            continue
        seen[key] = (group, answer)
        groups.setdefault(group, []).append({'prompt': [{'role': 'user', 'content': prompt}],
                                             'completion': [{'role': 'assistant', 'content': answer}]})
    parts = {'train': [], 'valid': [], 'test': []}
    by_label = {label: [] for label in sorted(LABELS)}
    for group, values in sorted(groups.items()):
        labels = {r['completion'][0]['content'] for r in values}
        if len(labels) != 1:
            raise ValueError('This classification recipe requires one label per source group.')
        by_label[labels.pop()].append(group)
    rng = random.Random(17)
    for label, keys in by_label.items():
        if len(keys) < 3:
            raise ValueError(f'{label} needs at least three source groups, one for each split.')
        rng.shuffle(keys)
        held = max(1, len(keys) // 10)
        parts['train'].extend(keys[2 * held:])
        parts['valid'].extend(keys[held:2 * held])
        parts['test'].extend(keys[:held])
    output.mkdir(parents=True, exist_ok=False)
    manifest = {}
    for name, ids in parts.items():
        values = [row for group in ids for row in groups[group]]
        data = ''.join(json.dumps(row, ensure_ascii=False) + '\n' for row in values).encode()
        (output / f'{name}.jsonl').write_bytes(data)
        manifest[name] = {'groups': ids, 'rows': len(values), 'sha256': hashlib.sha256(data).hexdigest(),
                          'labels': {label: sum(r['completion'][0]['content'] == label for r in values)
                                     for label in sorted(LABELS)}}
    (output / 'manifest.json').write_text(json.dumps(manifest, indent=2) + '\n')
    print(json.dumps(manifest, indent=2))


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--input', type=Path)
    parser.add_argument('--output', type=Path, default=Path('prepared'))
    args = parser.parse_args()
    if args.input:
        rows = [json.loads(line) for line in args.input.read_text().splitlines() if line.strip()]
    else:
        rows = []
        for label, cases in EXAMPLES.items():
            for i, case in enumerate(cases):
                for suffix in ['', ' Please choose the incident category.']:
                    rows.append({'group': f'{label.lower()}-{i}', 'messages': [
                        {'role': 'user', 'content': 'Classify as MEMORY, NETWORK, or SCHEDULER: ' + case + suffix},
                        {'role': 'assistant', 'content': label}]})
    prepare(rows, args.output)

if __name__ == '__main__':
    main()