Let me start with a picture.

Two students sit the same maths test. The first gets a big blank sheet. They can write out their working, make a mistake, cross it out and try again. The second gets a printed form with one tiny box, and nothing else. They must put the final number straight into the box.

Which student gets more answers right? Most teachers would bet on the first. Working it out on paper is how people avoid mistakes.

AI apps give language models that second form all the time. They ask the model to reply in a fixed data format, usually one called JSON (the next slide explains it), so that a program can read the answer.

Why not let the model write freely, and look for a line like "Answer: 42"? For one number, that works. But real apps often need many things back at once: a name, a date, a yes or no, a list. If the model words them a little differently each time, the program breaks. A fixed format gives every piece its own place.

So it is a very good habit for programs. This lesson measures what it costs in right answers.

Please read this slide slowly if any word is new. Every slide after it uses these words.

Model. Here, a language model: an AI that reads text and writes text, like the one behind ChatGPT.

JSON. A simple, fixed way to write data so a program can read it. It looks like this: {"answer": 42}. The part in quotes, answer, is called a field. The value after the colon, 42, is what the field holds.

Schema. A rule that says exactly which fields a JSON reply must have, and what kind of value goes in each. For example: "one field called answer, and it must be a whole number".

Enforced. Some model tools can force a reply to follow a schema. A model writes one small piece at a time, and at each step the tool blocks any piece that would break the rule. So if the schema says the answer is a whole number, the model must start writing digits at once.

Working. The steps a model writes before its final answer, like a student's rough work. Engineers often call this chain of thought.

Token. A small piece of text a model reads or writes, roughly three quarters of a word. More tokens means more time and more cost.

Temperature. A setting for how much a model varies its words. At 0, it always picks its most likely next word.

Model size. Model names end with their size. In llama3.2:3b, the 3b means about 3 billion numbers inside the model, learned in training. Bigger usually means more capable. Both models in this lesson are small.

The lab asks two models the same maths problems, in four different ways.

1. Free. "Work it out step by step, then end with a line: Answer: your number." The model can write as much working as it likes. A program then reads the number after "Answer:".

2. JSON, asked nicely. "Reply with only JSON, like this: {"answer": your number}." Nothing forces the model to obey. It might add extra words, and then a program cannot read it.

3. JSON, answer only, enforced. A schema forces the reply to be exactly one field: the answer. There is no room for working at all. This is the tiny box on the form.

4. JSON, working first, enforced. A schema forces two fields, in this order: first a "steps" field where the model writes its working, then the answer. The model still gets the fixed format, but it also gets room to think before it answers.

The last one is the idea to watch. It keeps what programs need, a fixed shape, and gives back what the model needs, space to work.

Why the order of the fields matters. A model writes from left to right, one token at a time. If the answer field came first, the answer would be written before any working existed, and the working after it could not change it.

The wording changed too. Each format came with its own one-line instruction, in quotes above. The free one invites working step by step; the answer-only one does not. So the lab changes the format and the line asking for it together. The limits slide comes back to this.

This lesson began as an accident in lesson 103.

In lesson 103, I first asked a small model to reply with just two numbers: its answer, and how sure it was. It got 2 of 200 easy problems right. Then I let it write out its working first. The same model got 197 of the same 200 right.

That was a big surprise. But a surprise from one test is only a hint. So this lesson tests it properly: two models, four reply formats, and a mix of easy and hard problems.

1. The problems. 70 maths word problems made by a small program, so every right answer is known. 40 are easy, like adding a few packs of books. 30 are hard, such as multiplying large numbers, or sharing things out with some left over.

2. The models. A small model, llama3.2:3b, and a slightly bigger one, qwen3:4b. Both run on a laptop, through ollama, a free tool for running models on your own machine. qwen3:4b can also write long hidden notes to itself before it answers, called thinking. The lab switched that off for every format.

3. The formats. Each model answers every problem in all four formats from the last slide. That is 8 answers per problem.

4. The mark. A program reads the number from each reply. If it matches the known answer, it is right. If the reply cannot be read at all, it counts as wrong, and the lab also counts it separately as unreadable.

Because every format answers the same problems, the lab can compare formats problem by problem.

Some problems were set aside. The hard problems come from the same small program as lesson 103. It made 10 problems about a shop selling tickets whose question can be read two ways, and the answer key took only one reading. A model that picked the other reading would be marked wrong unfairly, so they are left out, which leaves 70.

This is a real recording of the lab's report, printed on the laptop where the lab ran.

The recording uses short names for the formats. "free" is working, then "Answer:". "json_prompt" is JSON asked nicely. "json_answer" is JSON with one enforced answer field. "json_steps" is JSON with an enforced steps field first.

A few other words in it. The lab let each reply run to at most 3,000 tokens. "capped" means a reply hit that limit and was cut off; none did. "only first" and "only second" count the problems only one of two formats got right. "p" is how often a split that uneven would happen by luck; the "Is the Gap Real?" slide explains it. "5.7e-14" means a decimal point followed by 13 zeros and then 57: tiny.

Each number is right answers out of 70. Both models did best when they could write freely. With one enforced answer box, both collapsed, to 1 and 5. A steps field before the answer won most of the loss back, to 35 and 47, but it did not reach free for either model.

One more thing: qwen3:4b scored 0 when asked nicely. That is the strict reading, and the next slides show why it is not the whole story.

Look closer at the few right answers in the one-box format.

The one right answer for llama3.2:3b was a problem whose answer is 0. A shop has 64 stamps, sells all 64, and the question asks how many are left. It wrote 0.

qwen3:4b put 0 in the box on 35 of the 70 problems. When the box forced it to start with digits at once, it often wrote 0. That is a placeholder, not a real try. One of its 5 right answers was that same stamps problem.

So the one-box scores are even weaker than they look: the single right answer for llama3.2:3b was luck, and so was one of the 5 for qwen3:4b.

The "asked nicely" row needs a closer look, because the two models handled that request in opposite ways.

First, two ways a program can read a reply. A strict program reads the whole reply as JSON. If there is anything else in it, even one extra word, it fails. A lenient program searches the reply for the last {"answer": N} it can find, and ignores the rest. The results table used the strict way, because that is how most programs read JSON.

llama3.2:3b obeyed. Every reply was short JSON with no working, and only 1 of 70 was right. But look at the 11 replies a program could not read. They were not random junk. They looked like {"answer": 59 + 4 * 3}, where the star means times: the model wrote the calculation into the box and never worked it out. That is not valid JSON, because JSON wants a finished number there. Yet the calculation itself was right on 10 of those 11. The model knew how to set the problem up. It had nowhere to do the arithmetic.

qwen3:4b disobeyed. Remember it is built to think before it answers, and the lab switched that off. It thought anyway. On all 70 replies it wrote out its working and ended it with </think>, the marker that normally closes its hidden notes. Only then did it give the JSON.

A strict program could read none of those replies. A lenient program found the right answer on 70 of 70. So asking nicely did not get clean, readable JSON from either model. Only the enforced schema did.

The one box did not only hurt the hard problems. On the easy half, like adding a few packs of pencils, llama3.2:3b went from 40 of 40 when free to 1 with one box. qwen3:4b went from 40 to 4.

Here is the very first problem: Farah has 62 pencils and buys 3 packs of 3. With one box, llama3.2:3b wrote 67. With room to work, it wrote 3 x 3 = 9, then 62 + 9 = 71, which is right.

With working first, llama3.2:3b got 32 of 40 easy problems right, and qwen3:4b got 27 of 40. So on easy problems with working first, the bigger model got 5 fewer right than the small one. On the hard half it was the other way: qwen3:4b got 20 of 30, llama3.2:3b only 3. llama3.2:3b managed only 6 hard ones even when free, so there was little to win back.

Every format answered the same problems. So we can compare two formats problem by problem, and look only at the problems where they disagree.

Think of each of those problems as a coin flip. If the two formats were equally good, each would win about half of the flips. The "by luck" numbers below say how rarely a split at least this uneven would happen if they really were equally good.

Free against answer only: llama3.2:3b was right only with free on 45 problems, and right only with the box on 0. For qwen3:4b it was 65 to 0. A split like that happens by luck less than once in a million tries.

Free against working first: 12 to 1 for llama3.2:3b, which happens by luck about once in 293 tries. And 23 to 0 for qwen3:4b, which happens by luck less than once in a million tries.

So both gaps are real. The one box costs right answers. A steps field wins most of them back, but not all.

Working is not free. Every word the model writes costs time and money.

llama3.2:3b wrote about 141 tokens per reply when free, 7 with one box, and 100 with working first.

qwen3:4b wrote far more when free, about 886, because it wrote its thinking into the reply. That thinking was long, and every one of those replies stopped to check its own sums. Asked nicely, it thought anyway and wrote about 563. Working first held it to about 109.

So there is a real trade. Compared with one box, working first costs more tokens and gets more answers right. Compared with free, it cost qwen3:4b fewer tokens, but it also got fewer right.

The short working of qwen3:4b hides a problem. On 15 of the 70 problems, qwen3:4b left the steps field empty and wrote only an answer. It got 0 of those right. When it did write working, it got 47 of 55 right. A steps field gives the model room. It does not make the model use it.

This box holds the real results for all 70 problems on llama3.2:3b. For each format, one character per problem: 1 if the reply held the right answer, read the strict way a program would, 0 if not. Press Run to count them and compare two formats problem by problem.

Then find the line that starts with A, B =, a few lines down. Change the two names inside the quotes, for example to "json_steps" and "free", keep the quotes, and press Run again.

Maths problems only. Every answer is one number that needs working out. A task like "pull the customer's name out of this email" needs almost no working, and a fixed format may cost it little. This lab cannot say.

The wording changed with the format. The free instruction said "you may work it out step by step"; the answer-only one did not. Some of the gap may come from that sentence, not from the box.

qwen3:4b's free replies were thinking. Every one of them ended with a </think> tag: its hidden thinking leaked into the reply although the lab switched it off. So its free score of 70 is a thinking model at full stretch, which is more than just "free".

Two small models. Big models from companies like OpenAI and Anthropic, which sell them over the internet, may cope far better with one answer box. Measure the model you use.

One way of forcing the format. The lab used ollama's schema option, which blocks any token that breaks the schema. Other tools force formats in other ways, and may behave differently.

One run, at temperature 0. At temperature 0 a model picks its most likely words, so a rerun should give nearly the same replies. Lesson 97 showed that even then a rerun can move a little. And other problems or other wording could give different numbers.

1. Check the task. Does the answer need working out, like a sum or a decision, or only copying, like a name from a document? Here, on two small models, the one-box format hurt the first kind badly. Your model may differ, which is why step 3 matters.

2. Put the working first. If the task needs thinking, give the format a steps field before the answer field, in that order. Then look at real replies. Check the steps really come first, because some tools sort fields by name, which would put "answer" before "steps". And count empty steps fields: qwen3:4b left the field empty on 15 problems here.

   Or let it think somewhere else. Some models have a thinking mode that writes notes before the reply. Another option is two calls, meaning two separate requests to the model: a free answer first, then a cheap second request that copies the number into JSON. Test these the same way.

3. Test both ways. Run the same questions with and without the format, and compare right answers problem by problem.

4. Count what cannot be read. A model asked nicely may add words around its JSON, or write a sum where a number should go. Both happened here. Count those replies, and decide whether your program should look for the JSON inside the reply. If the steps can grow long, also count replies cut off by the length limit before the answer field; a cut-off reply is not valid JSON.

5. Count the tokens. Room to work costs tokens on every reply. Decide whether the right answers are worth it.