Think of a young football referee. She learned the job in one local league. The matches there were tidy: the same kind of teams, the same weather, the same small crowd, players who spoke her language. After a season she made almost every call correctly, and her coach was proud of her.

Then she is sent to referee a cup match far from home. The pitch is wet, the crowd is loud, two players shout at her in a language she does not know, and three things happen at once near the goal. Some of her calls are now harder. That is expected, and it tells her coach something useful: how good she is away from the matches she practised on.

Then something happens that the rule book simply does not cover. She checks the book, carefully, and finds nothing. A second referee, watching the same moment, makes the opposite call. Neither of them is careless. The book is missing a rule. No amount of practice fixes that. Only the league can fix it, by writing the rule down.

This lesson does the same to the two small models I trained in lesson 4. So far I have tested them on messages written in the same style as their training messages. Now I send them messages of six new kinds, and I compare them with two larger models that were only given the written rules. Both halves of the referee story happened: every model scored lower on the new messages, and one kind of message exposed a missing rule.

New cases are messages shaped unlike the ones a model was trained on, or unlike the ones you tested it on while you built it. People also call this a distribution shift: the kind of input changes, even though the task stays the same. A kind is one such shape. I chose six: long emails, two requests in one message, other languages (English mixed with Hindi, Bengali or Tagalog, or written wholly in Spanish or German), messages with decoy numbers (numbers that are not the order number), rare products with brands and sizes, and plain requests to cancel.

As in every lesson of this chapter, the task is a ticket: five fields (category, wants, order number, item and urgent) filled in by the house rules of lesson 2. An annotator writes the right ticket for a message. A field is gold when two annotators, working apart, filled it in the same way, and only gold fields are scored. A message is fully gold when all five fields are gold, and only those messages can score a whole ticket: all five fields right.

The leak line is the chapter's rule against testing on a copy of the training data: a new message whose closest training message has a cosine similarity (a score for how alike two texts are in meaning, usually between 0 and 1 here, where 1 means the same) of 0.80 or more is dropped. A rule gap is a case the house rules do not decide. Paired and unpaired comparisons are the two kinds from the prompting chapter's lesson 11: paired means the same messages, compared one by one; unpaired means different messages, where only totals can be compared.

Lesson 4 ended with a result that surprised me. On the chapter's 140 test messages, the fine-tuned 0.5B model got 93 whole tickets right of the 131 fully gold ones, the same as the prompted 14B, a model almost thirty times its size, and the fine-tuned 1.5B got 102. The fine-tunes had only a one-line prompt. The 14B had the full rules and a JSON schema.

But those 140 test messages were written like the training messages: short, one request each, plain English. The training messages run from 7 to 32 words, with a median of 17. A fair worry is that the fine-tunes learned the shape of those messages, not the job. Real customers do not write to a template. Some send a long email with their life story; some ask two things at once; some write half the message in their own language; some paste a tracking code where you expected an order number. A model that only works on tidy messages will fail quietly on exactly the messages that need the most care, and the ordinary test set will not warn you, because it is tidy too. So the question of this lesson is: on messages shaped unlike its training data, can the fine-tune still not be told apart from a prompted 14B, or does it fall behind? And a second question came out of the data: what can no model fix?

I wrote the design down in the description at the top of the lab file newcases.py: the six kinds, the four set-ups, the scoring and the 14 sign tests I would run. The record that this came before any message was written is the dated entry in the chapter's plan file, CHAPTER-PLAN.md (2026-09-28, the new-cases lab "designed before any message was written"). Then a writer who saw only the house rules and the list of kinds wrote 90 messages, 15 of each kind. The writer never saw the training set or the test set. The messages were written by an AI model and labelled by two AI models (annotators H and I), as with all of this chapter's data; they are not real customers.

Each kind tests something the training data did not show. Here is one real message of each.

Long messages are emails with a greeting, a story, sometimes a quoted earlier reply and a signature with a phone number. The request is buried in the middle. Two messages ask for two things; the rules say to choose the action asked for first. Mixed messages are in other languages: 9 mix English with Hindi or Bengali (written in Latin letters) or Tagalog, and 6 are written wholly in Spanish or German. The design asked for English mixed with another language; the writer wrote 6 with no English at all, and I kept them, because customers do that too. Decoy messages hold other numbers next to the order number, or instead of it: a phone number, a price, a date, a tracking code, an invoice number. Rare messages are about unusual products, with brands, sizes and adjectives that the item rule says to strip. Cancel messages ask to cancel an order that has not shipped, with no refund asked.

The lab uses the same four set-ups as the chapter's test runs, with nothing changed, so the two sets of results can be put side by side.

The two fine-tunes are lesson 4's LoRA adapters on Qwen2.5 0.5B and 1.5B, run in mlx, the tool that trained them. They get only the one line they were trained with: "Turn the customer message into a support ticket as one line of JSON." A LoRA adapter is the small add-on lesson 4 trained beside a frozen base model. They choose the most likely next token (a piece of a word, the unit a model reads and writes) every time, which is called greedy decoding, and may write up to 80 tokens. Why only one line? Because that is how they were trained, in lesson 4: the house rules were never shown to them as text. Everything they know about the rules, they learned from 500 labelled examples. That is exactly what this lesson tests: whether rules learned from examples carry over to messages unlike the examples.

The two prompted models are qwen2.5:7b and qwen2.5:14b in Ollama. They get the full house rules as the system message and the chapter's JSON schema (lesson 3: a description of the allowed shape of a reply, which Ollama enforces as the model writes). They run at temperature 0, which means always take the most likely next token, with seed 1, a fixed starting number for any random choice, so a rerun gives the same reply.

The grader is the chapter's own. It reads each reply as JSON and compares each field with the gold value, after putting text in lower case. Every reply from all four set-ups was valid JSON holding the five fields, so no score here comes from a reply the grader could not read.

Before scoring, the lab found each new message's closest training message, using the same embedding model as the chapter's other leak checks (nomic-embed-text). An embedding turns a text into a list of numbers so that texts with similar meaning get similar numbers, and cosine similarity compares two such lists.

Seven of the 90 were at or over the leak line of 0.80: two decoy, two cancel, two long and one two-request message. Reading the pairs, most are the same situation in new words: a price difference on pillows, a refund for a returned rug, a lamp that arrived broken. The writer never saw the training data, so these are not copies; a shop's customers simply ask similar things. The rule drops them anyway, because a close match could let a fine-tune score by memory rather than skill. Dropping them costs a little test data; keeping them could make a fine-tune look better than it is on exactly the question this lesson asks. When in doubt, a test set should lean towards being too strict.

That leaves 83 scored messages. The closest of them to any training message was a plain cancel, "Please cancel order 41822, I no longer need it", at 0.791, just under the line. The mixed messages were the furthest from training, with a median best match of 0.574: an embedding of a sentence written wholly in Spanish is far from any English one.

Before looking at any model, look at the people, or here the AI annotators, who wrote the answers. Their agreement decides what can be scored at all. A model can only be marked right or wrong against an answer that someone settled, and a field the annotators never settled cannot be marked at all, however good or bad the model's reply.

On the five kinds other than cancel, H and I agreed on all five fields for 57 of 75 messages. Most of their splits were small: whether the item is "table lamp" or "lamp", "frying pan" or "pan". The two-request messages were the least settled of these, fully agreed on 8 of 15, because deciding which request comes first is itself a judgement.

On cancel, the picture is completely different. H said delivery for all 15 cancel messages. I said billing for all 15. They agreed on the category of none. They were not careless: on the other four fields of the same messages they agreed almost everywhere. They read an unclear rule two ways, and each kept to their own reading. That is the useful part. When annotators split here and there, a message is usually just hard to read. When they split the same way on every message of a kind, they are following two different rules, and the written rules allow both. Here it was 15 of 15 one way and 15 of 15 the other.

The same split appeared outside the cancel kind: 5 messages of other kinds also split between delivery and billing, and every one of them involves cancelling something, such as "please cancel order 40277 and also how do I change the password". So of the 83 scored messages, 53 are fully gold, and none of the 13 scored cancel messages is among them. Cancel has no whole ticket to score, though its other four fields can still be scored.

Here is the headline, on the 53 fully gold new messages, which all four set-ups answered.

The fine-tuned 0.5B got 22 whole tickets right, the fine-tuned 1.5B 21, the prompted 7B 24 and the prompted 14B 25. The four are close together, and all four got fewer than half right.

Because all four set-ups answered the same messages, this comparison is paired, and the sign test from earlier lessons applies. It looks only at the messages where two set-ups disagree, one right and one wrong, and asks how likely a split that uneven would be if only luck were at work. That chance is the p value. Against the 14B, the fine-tuned 1.5B was right alone on 5 messages and wrong alone on 9: p 0.4240. The 0.5B: 6 against 9, p 0.6072. Neither can be told apart from luck.

It helps to see what those numbers mean in messages. Of the 53, the fine-tuned 1.5B and the 14B were both right on 16 and both wrong on 23. They disagreed on only 14 messages, and the split of those 14 was 5 to 9. Suppose the two were equally good. Then a split of 14 messages would often be uneven just by chance: 5 to 9, or something more lopsided, would happen about four times in ten. That is what p 0.4240 means. So a gap of four tickets on 53 messages says very little.

I declared 14 sign tests in the design (the record is the CHAPTER-PLAN.md entry above): each fine-tune against the 14B, per kind and over all messages. With 14 tests, the Bonferroni line (the usual 0.05 divided by the number of tests, so that running many tests does not make a lucky result easy to find) is 0.00357. The smallest p of the 14 is 0.4240, nowhere near it. Two of the 14 turned out empty once the labels came in, because cancel has no fully gold message; I keep them in the table because they were declared.

Every set-up got a smaller share right on the new messages than on the chapter's 140 test messages. This comparison needs care, because the two sets are different messages.

On the chapter's 140 test messages, the fine-tuned 0.5B got 93 of 131 fully gold messages right, 71%. On the new messages it got 22 of 53, 42%. For the fine-tuned 1.5B it was 102 of 131 (78%) and 21 of 53 (40%); for the prompted 7B, 80 of 131 (61%) and 24 of 53 (45%); for the prompted 14B, 93 of 131 (71%) and 25 of 53 (47%).

It is tempting to say "the 0.5B dropped from 93 to 22". That sentence is wrong. No message is in both sets. The new messages were written by a different writer, labelled by different annotators, and chosen to be hard. The two numbers describe two different exams, not the same exam taken twice. What you can say is that every set-up scored lower on this new set than on the 140.

The comparison is unpaired, so the sign test does not apply. The prompting chapter's lesson 11 used Fisher's test for this: it asks how often two groups of this size would split this unevenly if both had the same chance of a right answer. I ran it after seeing the data, so read it as a description. For the two fine-tunes it gives p 0.0004 and below 0.0001; for the 14B, 0.0037; for the 7B, 0.0704. So three of the four scored lower by more than luck would usually give, and the 7B's lower score cannot be told apart from luck. The fine-tunes had further to fall, because they started higher on the 140. But whether they fell further than the 14B is a question about two different sets, and on the same 53 new messages, the paired test above could not separate them.

The two sets also differ in what they contain. The 131 include cancel requests whose category the annotators happened to agree on, so they were scored: te057 and te059 as delivery, te058 and te125 as billing. The 53 new ones include no plain order cancel at all, because the new annotators split on every one.

The totals hide how each kind went. The fully gold messages per kind are few, from 7 to 13, so read these counts as a picture, not a measurement.

The long emails went best for every set-up: 8, 7, 8 and 9 of 12. That surprised me most. The fine-tunes had never seen a message longer than 32 words, and here they read emails of up to 227 words and found the request in the middle. The two-request messages went worst: 1, 2, 3 and 2 of 7. The mixed kind was hard for everyone too: 2, 3, 3 and 4 of 11. On decoys the prompted models did a little better (6 and 6 against 5 and 4), and on rare products the fine-tunes did (6 and 5 against 4 and 4). None of these per-kind differences can be told apart from luck.

Across the 53, all four set-ups got the same 10 messages right, and all four got 17 wrong. That second number matters: on a third of the messages, no set-up at all wrote the whole ticket. The 17 are not spread evenly. Seven are mixed messages, four are rare products, three are two-request messages, two are decoys and one is a long email. On mixed messages, 7 of the 11 fully gold ones defeated every set-up, which says the kind itself, or its gold, is hard, not that one kind of model is weak. The next slides read the replies to find out why.

Field by field, item and wants were the weakest fields on the new messages, for the fine-tunes and the prompted 14B alike. On the 140, the fine-tuned 1.5B had wants right 99% of the time; here, 84%. The 14B had item right 91% of the time on the 140 and 78% here. Again, these are two different sets, so these shares describe the sets; they are not a measured change on the same messages.

The rules say: "A phone number, price or date is never an order number." Decoy messages test that sentence, and also numbers it does not mention.

Where the right order number was null, the fine-tunes wrote a number anyway 5 times each, the 7B 3 times and the 14B once. Reading them, the fine-tunes took a tracking code, a parcel number, an item code, an invoice number and a customer number. The 7B once took the date "19/09". The 14B was caught only by the invoice number, which all four took.

Here is one possible reason, and it is a guess. In the training data, 280 of the 500 messages hold a digit, and 238 of those have an order number. So in the examples, a number in the message was usually the order number, and the fine-tunes may have learned "copy the number". The rule sentence names phone numbers, prices and dates, but not tracking codes, invoices or customer numbers. The 14B, reading the full rules, seems to have taken the idea behind the sentence; the fine-tunes never read that sentence at all, only examples of it. None of them ever took a phone number, a price or a date as the order, apart from the 7B's one date.

The decoy kind was the only one where the prompted models were ahead on the order number field: 12 of 13 for the 14B against 8 of 13 for each fine-tune.

What would I change? The rule itself is the first place to look. One more sentence, "a tracking code, parcel number, invoice number or customer number is never an order number", would cost nothing for the prompted models: they would read it on their next reply. For the fine-tunes, a sentence does nothing, because they never read the rules. They would need training examples that hold decoy numbers, labelled with a null order number, and a new training run. This pattern, a rule fix that is cheap for a prompt and costly for a fine-tune, comes back on the cancel slide.

The item rule asks for "the product the message is about, as a short common noun". For a message written in Spanish or German, the annotators wrote the English noun: "blanket" for "manta", "kettle" for "Wasserkocher".

All four set-ups wrote "manta" for the Spanish blanket. For the German kettle the prompted models wrote "wasserkocher" and the fine-tunes wrote pieces of it, "kocher" and "wasserkok". For "Bettwäsche" the answers were "bath", "bedsheet", "bettwäsche" and "bedsheet". The item field on mixed messages was wrong 6, 5, 7 and 5 times of 13 gold.

This is a mistake shared by all four, not a fine-tune problem. And one could argue about the gold itself: the rules never say directly that the item must be in English. The annotators agreed on English, so English is gold here. This is a smaller version of the cancel gap: the model is asked to follow a rule that is not written down. The fix is again a sentence: "write the item as an English noun, whatever language the message is in". Until the rules say it, a model that keeps "manta" has written a short common noun, as the item rule asks, and has broken only a rule nobody wrote.

What the mixed messages did well: the order number. The fine-tunes read "Pedido 83012" and "Bestellung 27459" as order numbers every time on these messages, and the order number field was right for all four set-ups on all 15 mixed messages.

The rules say: "If the message asks for two actions, choose the one it asks for FIRST." A two-request message tests whether the model follows that sentence.

I read the 15 two-request messages, from their text only, and wrote down the first and the second action each one asks for, as wants values; the file is data/newcases/two_actions.json, with the rule I used. I did this after the results were known. Then the report counts, for the 14 scored ones, whether each reply's wants was the first action, the second (or a later one), or neither. The fine-tunes took the first action 11 times each and the later one once each. The prompted 7B took the first 9 times and a later one 4 times; the 14B the first 8 times and a later one 5 times. "Where is my order 60419? ... If it's not coming this week, cancel it and refund me": the first request is information, and the 7B wrote cancel, the 14B refund.

The one message all four sent to the later action was "Please send me the return label for the duvet. Once that's done, can you let me know how long the refund usually takes?": all four wrote information. The 14B's later choices were not always the more forceful action: it chose information over stop ("Stop texting me about reviews. I also want to know why order 71114 still hasn't shipped") and over the label. Its replies suggest a habit of taking the later request, but that is a pattern I read after the results, on 14 messages and one run, and no reason for it is supported by this data. The later request also pulled the 14B's order number: on two messages where the order number belongs to the second request, it wrote null. If your customers often ask two things at once, the rule "choose the one asked for first" is worth testing on its own: write twenty such messages, mark the first request before any model runs, and count.

Two replies from the fine-tuned 1.5B were a new kind of broken. On one two-request message it added a sixth key, other, and on another it wrote extra keys after the five, repeating urgent three times. Both were still valid JSON, so the grader read them, taking the last value of a repeated key. The prompted models, held to the schema, never did this here. It is the same weakness lesson 9 found: without a schema, a fine-tune can wander out of the shape.

The item rule says: no colour, size, brand, quantity or adjective. The rare products test it with names like "the 25cm round rattan banneton with linen liner".

The prompted 7B and 14B had the item right on 12 and 11 of the 15 rare messages; the fine-tunes on 8 and 9. The mistakes differ in kind. All four wrote "bird bath fountain" where the annotators wanted "bird bath"; that one is arguable, since a fountain is part of what the product is. The fine-tunes' own mistakes are stranger: the 1.5B wrote "galvanised steel" for the composter and "detect" (a word from the model name "V15 Detect") for the vacuum. These look like a model grabbing a word from the product's description instead of the product itself, not like a model that kept too much.

The prompted models were more likely to keep the right noun inside a longer name, 4 times each over all 83 messages, against 2 and 3 for the fine-tunes. On whole tickets, the rare kind was where the fine-tunes did best against the 14B: 6 and 5 against 4 of 13.

The long emails deserve a second look, because they are the kind I expected the fine-tunes to fail. They did not. Their misses were few and mostly on wants. On a password reset email the fine-tunes wrote "other" where the answer is "change". That also happened on a short password message in Bengali and English and on a two-request one. One possible reason is in the training data. I found the training messages about signing in by a keyword search (password, passcode, reset, log in, login, log-in, sign in, signin, sign-in, locked out, two-step, verification code): 12 messages, labelled change 4 times, other 5 times and information 3 times. The fine-tunes may have learned a split habit from split examples.

Now the kind that no model could score on. Here is the whole story of one message.

"Please cancel order 41822, I no longer need it." Which team gets this ticket? The rules give four categories. Delivery covers "shipping, couriers, tracking and parcels that have not arrived". Billing covers "charges, payments, invoices, prices and refunds of money", and the boundary rule adds that "the refund of a cancelled order" is billing. But this customer asks for no refund. The order has not shipped, so it is not a delivery problem either. Nothing in the rules decides it.

The annotation files hold labels, not reasons, so here is my guess at the two readings. One reading is delivery (the order is stopped before it ships, which is the warehouse's and the courier's business); the other is billing (stopping an order stops a payment). Annotator H chose delivery and annotator I billing, on all 15 cancel messages without exception. Lesson 2 found exactly this hole when two other annotators relabelled the training data, and I left it in on purpose. Here it is, measured: 0 of 15.

The four set-ups did not split. The fine-tuned 0.5B wrote billing on 11 of the 13 scored cancel messages, the 1.5B on 12, the 7B on 11, the 14B on 12. On the other four fields they did well: wants was right on all 13 for all four, and so was the order number.

A model can learn the rules, from examples or from reading them. It cannot write a rule that is missing. When two careful annotators split 15 to 15, a model that picks one side is right by one reading and wrong by the other. No amount of training data, model size or prompt wording changes that, because the disagreement is not in the model. It is in the rules.

The fix is a decision, not a model. The shop decides: a plain cancel goes to, say, delivery, because the warehouse team stops the parcel. Someone writes that down as one sentence in the rules. Then the two routes differ in cost. For the prompted model, the new sentence goes into the guide it reads, and the next reply can follow it. For the fine-tune, the rule lives in its training labels, so every training row the rule touches must be relabelled, and the model trained again and tested again. Here the prompt is cheaper to change, and that is one real cost of fine-tuning that this chapter had not measured before. It is easy to miss, because on the day you train a fine-tune its rules look settled. Rules change for ordinary reasons: a new team is formed, a product line is added, a lawyer asks for privacy requests to go somewhere else. Each change is one sentence for a prompt, and a relabel-and-retrain cycle for a fine-tune, with a fresh test at the end. If you expect your rules to change often, count that cost before you choose.

Either way, the test set needs the rule too. Until the annotators agree, the cancel messages cannot score a whole ticket, for any model.

This script sends five of the new messages, one per kind apart from long, to a fine-tuned model and a prompted model in Ollama, each set up the way the lab ran it: the fine-tune gets the one line it was trained with and no schema; the prompted model gets the full rules and the schema. It prints each reply and which agreed fields it got wrong. It uses only Python's standard library. (A comment in the script says the five messages are one per kind; it means one per kind other than long.)

Before you run this lab. It needs Ollama running on your own computer, with qwen2.5:14b (ollama pull qwen2.5:14b, about 9 GB) and the fine-tuned 1.5B you imported in lesson 9 as ft-q15-f16 (its 16-bit import: the model merged with its adapter and loaded into Ollama at 16 bits per weight). If you have not set Ollama up yet, the lab setup guide shows how to install it and check that everything works, on macOS, Windows or Linux. A smaller machine can use qwen2.5:7b and ft-q05-f16 instead: python newcases_demo.py ft-q05-f16 qwen2.5:7b. I do not give a running time, because another lab was using the same machine while mine ran.

If you use Windows or Linux: the fine-tunes were trained and scored in mlx, which is built mainly for Apple silicon, and everything here ran on a Mac. Lesson 9 shows how to move a fine-tune into Ollama, which runs everywhere; to train one without mlx, Hugging Face's PEFT library on a free Colab GPU is the usual route. I have not run that path for this chapter, and nothing here claims the Mac's numbers hold elsewhere.

"""Five new kinds of message, sent to a fine-tuned model and to a prompted one, each set up the way the lab ran it.

This is the lab of lesson 10 of the fine-tuning chapter, made small. It needs Ollama running on your computer,
the fine-tuned 1.5B you imported in lesson 9 (ft-q15-f16) and qwen2.5:14b (ollama pull qwen2.5:14b).
It uses only Python's standard library:
    python newcases_demo.py                          # ft-q15-f16 and qwen2.5:14b
    python newcases_demo.py ft-q05-f16 qwen2.5:7b    # any models; a name starting "ft-" gets the fine-tune's set-up
"""
import json
import sys
import urllib.request

FIELDS = ["category", "wants", "order_number", "item", "urgent"]

# The fine-tunes: the one line they were trained with, no schema.
SHORT = "Turn the customer message into a support ticket as one line of JSON."

# The prompted models: the house rules, word for word as in TICKET-GUIDE.md, plus the JSON schema.
RULES = """Turn one customer message into one ticket, a JSON object with exactly these five fields.

## category (one of four, same definitions as the prompting chapter)
- billing: charges, payments, invoices, prices and refunds of money.
- delivery: shipping, couriers, tracking and parcels that have not arrived or arrived damaged.
- returns: sending an item back, exchanges and the return process.
- account: signing in, passwords, profile details, privacy and emails from us.

Boundary rule: a refund for an item the customer
has already sent back, or is sending back, is **returns**, because the refund is the last step of the return.
Money that is not tied to sending an item back (a wrong or double charge, an invoice, a price, the refund of a
cancelled order) is **billing**. A parcel that never arrived or arrived damaged stays **delivery**, even when the
customer asks for their money back.

## wants (one of seven): what the customer is asking us to DO
- refund: give money back.
- replacement: send the same item again, or swap it for another (an exchange).
- cancel: stop an order, subscription or payment before it happens or continues.
- change: edit details we hold (address, email, name, phone, password, payment card, delivery slot).
- stop: stop messages from us (emails, texts, calls, newsletters).
- information: they ask a question or for a status, and ask for no other action.
- other: none of the above (a complaint with no request, praise, deleting data, something else).
If the message asks for two actions, choose the one it asks for FIRST.

## order_number (string or null)
The order number exactly as digits, without "#", "order" or spaces ("order #4471" -> "4471").
null when no order number is written. A phone number, price or date is never an order number.

## item (string or null)
The product the message is about, as a short common noun: lower case, singular, no colour, size, brand,
quantity or adjective ("two blue ceramic mugs" -> "mug", "my new Sony headphones" -> "headphones",
"a pair of trainers" -> "trainers"). Keep nouns that are always plural in English (headphones, trainers,
jeans, scissors). null when no product is named ("my order", "the parcel", "it" are not products).

## urgent (true or false)
true only when the customer states a time limit or deadline, or says urgent, asap, today, tonight,
right now, or immediately. Anger alone is not urgency.

## Output
One line of JSON, keys in this order:
{"category": "...", "wants": "...", "order_number": "..." or null, "item": "..." or null, "urgent": true/false}"""

SCHEMA = {"type": "object", "properties": {
    "category": {"type": "string", "enum": ["billing", "delivery", "returns", "account"]},
    "wants": {"type": "string", "enum": ["refund", "replacement", "cancel", "change", "stop", "information", "other"]},
    "order_number": {"type": ["string", "null"]}, "item": {"type": ["string", "null"]}, "urgent": {"type": "boolean"}},
    "required": FIELDS}

# Five of the 90 new messages, one per kind, with the ticket both annotators agreed on ("?" = they did not agree).
MESSAGES = {
    "nc_two_15": ("Where is my order 60419? It was meant to come Monday. If it's not coming this week, cancel it and refund me.",
                  ["delivery", "information", "60419", None, True]),
    "nc_mixed_07": ("Buenos días, el paquete con la manta llegó mojado y la manta tiene manchas. Quiero una nueva por favor. Pedido 83012.",
                    ["delivery", "replacement", "83012", "blanket", False]),
    "nc_decoy_02": ("Tracking code JD014472896GB has said 'out for delivery' since Tuesday. Postcode is BS7 8QA. When will my parcel actually get here?",
                    ["delivery", "information", None, None, False]),
    "nc_rare_08": ("The solar-powered copper-finish bird bath fountain with LED ring I bought was charged at £89 but listed at £69. Refund the difference.",
                   ["billing", "refund", None, "bird bath", False]),
    "nc_cancel_01": ("Please cancel order 41822, I no longer need it. Thank you.",
                     ["?", "cancel", "41822", None, False]),
}


def ask(model, message):
    fine_tune = model.startswith("ft-")
    body = {"model": model, "stream": False,
            "messages": [{"role": "system", "content": SHORT if fine_tune else RULES},
                         {"role": "user", "content": message}],
            "options": {"temperature": 0, "seed": 1, "num_predict": 120} if fine_tune else
                       {"temperature": 0, "seed": 1, "num_ctx": 8192, "num_predict": 120}}
    if not fine_tune:
        body["format"] = SCHEMA          # the prompted models ran with the schema, as in the lab
    req = urllib.request.Request("http://localhost:11434/api/chat", data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req, timeout=900).read())["message"]["content"]


def wrong_fields(reply, gold):
    try:
        got = json.loads(reply)
    except ValueError:
        return "not JSON"
    bad = [f for f, g in zip(FIELDS, gold) if g != "?" and str(got.get(f)).lower() != str(g).lower()]
    return "all agreed fields right" if not bad else "wrong: " + ", ".join(bad)


for mid, (message, gold) in MESSAGES.items():
    print(f"{mid}: {message if len(message) <= 70 else message[:67] + '...'}")
    for model in sys.argv[1:] or ["ft-q15-f16", "qwen2.5:14b"]:
        reply = ask(model, message)
        print(f"  {model}: {reply}")
        print(f"  {' ' * len(model)}  {wrong_fields(reply, gold)}")

The report lives in scripts/labs/finetune/newcases_report.py. It reads only stored files: the 90 messages, the two annotations and the key that hides which message is which, the gold, the similarity file, the four runs, the declared sign tests, the training data and the chapter's runs on the 140. It calls no model. A json mode writes the numbers to results/nc-report.json, which the figures read.

Before it prints anything, it checks the files against each other. The gold must be exactly what the two annotations give; every stored grade is graded again against the gold; each run's rows must be the scored messages in order; and the 14 sign tests are computed again and must match the ones the lab stored. If anything differs, the report stops.

The demo mode checks the student script's stored run: its prompt, rules and schema must be the lab's, its messages and gold must match the files, and the prompted model's replies must equal the stored ones. The box mode writes the playground below and checks that it reproduces every field of every reply's score.

What came after I saw the data: the per-field tables, the Fisher tests between the two sets, the counts of kinds of mistake, my reading of which action each two-request message asks for second, the cancel counts from the training data, the demo's five messages and every example in the figures. The kinds, the four set-ups, the leak line, the scoring and the 14 sign tests were fixed before the run, in .

This box has no model in it. It holds the 83 scored new messages, their kind, the gold ticket (a ? where the annotators did not agree) and the four set-ups' real replies. A reply that is JSON with the five keys in order is stored as a ticket; any other reply is kept as its text, and the box reads it as JSON when it scores it.

As it is, the box prints the whole tickets of all four set-ups (22, 21, 24 and 25 of 53), then the decoy messages field by field, then one decoy message with every reply: the kettle with an item code, a date and a tracking number, where both fine-tunes wrote "kt-2210" as the order number, the 7B "19/09", and only the 14B null.

Try score('long') or score('two') for one kind. Compare fields('mixed') with fields('decoy'): on mixed messages the item field is where every set-up loses, and on decoys it is the order number, for the fine-tunes above all. Try fields('cancel') to see category scored on nothing, because no cancel message has a gold category, while the other four fields still score. Try show('nc_cancel_01') to see four set-ups agree on billing where the gold says ?, and show('nc_mixed_07') for the blanket that all four called "manta".

The prompts. SHORT is the one line the fine-tunes were trained with, word for word. RULES is the house rules, word for word as in TICKET-GUIDE.md, which the prompted models read as their system message. SCHEMA is the chapter's JSON schema.

The messages. MESSAGES holds five of the 90 new messages, keyed by their ids, each with the ticket both annotators agreed on. A ? marks a field they did not agree on; the script does not score it.

The request. ask decides the set-up from the model's name. A name that starts with ft- is treated as a fine-tune: the one-line prompt, no schema, temperature 0, seed 1, at most 120 tokens, as in lesson 9. Any other name gets the rules, the schema and the lab's context window of 8,192 tokens (the most text the model reads at once). The request goes to Ollama's local chat address, and the reply's text comes back.

The check. wrong_fields reads the reply as JSON and lists the agreed fields that differ from the gold. It is a small version of the lab's grader; it compares text in lower case, as the grader does.

The loop. For each message it prints the start of the message, then each model's reply and what it got wrong. Read the replies side by side: every finding in this lesson started that way.

List the kinds. Look at real messages, or ask the people who read them, and list the shapes your training data does not have: long, mixed, several requests, strange products, numbers that are not what they seem. Write the list down before you score anything.

Have someone else write each kind. The writer should see the rules, not the training data or the test set. Aim for the same number of each kind, so no kind hides inside a total.

Label blind, twice. Two people label every message without seeing each other's work or the kind. Keep only the fields they agree on. Count where they split, kind by kind.

Drop leaks. Compare each new message with the training data and drop anything too close. Report how many you dropped and why.

Compare set-ups on the same messages. Put your fine-tune next to the best prompted model on the same messages, per kind, with a sign test declared in advance. Never compare a score on these messages with a score on other messages as if it were the same test.

Fix the rules where the labellers split. A split is a finding about your rules, not noise. Write the missing rule, relabel what it touches (the training data too), and only then test again.

Before you ship any fine-tune. A fine-tune learned from examples of one shape. Its test set, if it came from the same writer or source, has the same shape. A new-cases test is the only way to see how it does on the inputs that the examples never covered. Here, it held; on another task it may not.

When the fine-tune is replacing a prompt. The comparison that matters is on the same messages. On the 140 test messages, the fine-tuned 0.5B could not be told apart from the 14B; on the new kinds, it still could not. The second result tells you the first was not only about tidy messages.

When your rules are still changing. A prompted model follows a rule change with one edited sentence; a fine-tune needs relabelling and retraining. If your labellers still split often, fix the rules before you fine-tune, or you will train the split into the model, as the cancel labels were trained into these fine-tunes.

Not as a replacement for the ordinary test set. New cases tell you about shapes you chose to test. The ordinary test set tells you about the messages you usually get. You need both.

Not with a handful of messages per kind if you want a verdict. With 7 to 13 fully gold messages per kind, this lab can show patterns, not prove differences. For a decision that matters, write more of the kinds that matter most to you.

Not as a reason to distrust every fine-tune. The result here is reassuring: two small fine-tunes, trained on 500 short messages, could not be told apart from a prompted model almost thirty times larger, on messages unlike any they were trained on. A new-cases test is how you earn that confidence, not a ritual for finding fault.

One run of each set-up. Every set-up answered every message once. Lesson 5 showed that retraining with a different seed flipped 15 to 21 tickets on the 140, so small differences here mean little.

53 fully gold messages. That is enough to see that the four set-ups are close, not enough to rank them. Per kind there are 7 to 13, which is why no per-kind difference can be told apart from luck.

Two fine-tunes and two prompted models, on one task. Qwen2.5 0.5B and 1.5B, trained once each in lesson 4, against Qwen2.5 7B and 14B. I make no claim about other sizes, other model families or other tasks.

The design was declared; the reading was not. The kinds, the set-ups, the leak line and the 14 sign tests were written down before any message existed; the record is the dated CHAPTER-PLAN.md entry. The per-field tables, the Fisher tests, the kinds of mistake, my reading of the two-request messages and every example were chosen after I saw the replies. Read those as illustrations.

The gold can be argued with. The annotators agreed on English nouns for foreign words and on billing for a question before buying, where the rules say nothing. Agreement makes a field scoreable; it does not make it the only reasonable answer.

The data was written and labelled with an AI model's help, as in every lesson of this chapter. Real customers write differently, and real annotators may split in other places.

If you have a fine-tune, or are thinking of training one, you can start this week. Write down the kinds of message your real users send that your training data does not show. Ask a colleague to write ten of each from your rules alone. Have two people label them without talking. Then score your fine-tune and your best prompt on the same messages, kind by kind, and read every reply where they differ. Look hardest at the places where your two labellers split: that is where your rules need a sentence, and no model can write it for you.

The chapter's wrap brings it together: when fine-tuning was the right decision in these labs, what it cost, and what it could not do.