When a new colleague joins a team, you can explain the job in words, or you can work next to them through a few real cases together. Most people learn faster from the cases. The words say what to do; the cases show what "done" looks like, including all the small details nobody thought to write down.

A language model can be taught the same way inside a prompt. Before the real question, you include a few finished cases: a message, then its right answer, written in exactly the form you want back. These are called examples, and a prompt that contains them is often called a few-shot prompt.

Lesson 1 added two examples to a prompt that was already perfect, so it could not tell whether they helped. This lesson tests examples properly, on a prompt that still makes mistakes, and on two models. It asks three questions: does adding examples help, does it matter which ones you choose, and does it matter what order they are in?

Example. One finished case inside the prompt: a message, then the answer you would want for it, written in the same format you want the model to use.

Zero-shot. A prompt with no examples at all, only the instruction. In this field a "shot" means one example.

Few-shot. A prompt with a handful of examples, often one to five.

Order. Which example comes first in the prompt and which comes last, just before the real question.

Choice. Which messages you picked to be the examples. Two sets can cover the same categories and still be quite different messages.

One more idea from earlier lessons matters here: the model does not learn anything permanent from the examples. It only reads them, as part of the text of this one request, and they change which next token is most likely. Remove them from the next request and they are gone.

To see whether examples help, the prompt must still make mistakes without them. So this lab uses lesson 1's prompt C: the instruction naming the four categories (billing, delivery, returns, account) and the format rule ("Answer with the category word only, in lower case, and nothing else"), but not the context lines that made prompt D perfect. The test set is the same 40 customer messages, 10 per category.

I wrote 10 new messages to use as examples. None of them is one of the 40 test messages, and the lab checks that before it runs; an example that is also a test message would make the test meaningless. The check only catches exact copies, though. A few examples are close in meaning to a test message (for instance "Please stop sending me newsletters" and the test message "How do I turn off your marketing emails?"), so some scores may be a little flattering. Each example is written exactly like the question: "Message:", the text, then "Category:" and the answer.

The lab tries six versions of the prompt:

• no examples: prompt C alone.
• set A: one example per category, in the order billing, delivery, returns, account.
• set A, reversed: the same four, in the order account, returns, delivery, billing.
• set B: four different examples, one per category, in the same order as set A.
• three billing and three account: three examples that all share the same answer.

Start with llama3.2:3b and no examples. It got 20 of 40 right, and its mistakes were not spread out. It answered "returns" 30 times, though only 10 messages were about returns. Seven billing messages, five delivery messages and eight account messages all came back as "returns".

This is worth noticing, because it is typical of what goes wrong. A model with an unclear task often keeps giving one answer. The fallback here happened to be "returns"; with another model or another prompt it could be any of the four. The score, 20 of 40, hides this. Counting which answers came back shows it immediately.

qwen2.5:3b got 36 of 40 on the same prompt, with a different pattern of mistakes: three account messages answered "billing", and one billing message answered "returns". That is the same prompt C that lesson 1 measured, with the same result.

Adding set A, one example per category, took llama from 20 to 33 right. The "returns" habit disappeared: it answered returns exactly 10 times. One possible reason: with one example of each category in front of it, the model saw all four answers being used.

But the mistakes did not vanish. They moved. With set A, seven account messages came back as "billing". None of the account messages had been answered "billing" before. The examples fixed one pattern of mistakes and created another.

On qwen, set A took the score from 36 to 39. A model that already does the task well has less room to improve, and examples added only a little.

Now the same four examples, in reverse order: account first, billing last. Nothing else changed, not one word of the examples or the rules. Llama's score fell from 33 to 28.

And again, the mistakes moved. In set A's order, the seven mistakes were all account messages answered "billing". In the reversed order, the account messages were all right, but five delivery messages and four billing messages went back to "returns", and three billing messages were answered "account".

It is tempting to explain this with a simple rule, such as "the model copies the last example". The data does not fit that rule. In the reversed order the last example was billing, yet billing messages were answered wrongly more often, not less. Nor does "copies the first example" hold up: set A over-answered billing (17 times) and set A reversed over-answered account (13 times), but set B, also with billing first, answered billing exactly 10 times. What the data does show is simpler and more useful: on this model and this task, the order of the examples changed both the score and which messages were wrong. On qwen, both orders gave 39.

Why would the order of four examples change an answer? The previous chapter's first lesson gives the tool to look. At every step a model gives odds for every possible next token, and at temperature 0 it takes the most likely one. The lab asked Ollama for those odds on the first token of the answer, for three messages, under three of the prompts.

For "Please delete my account and all my data", with no examples, llama gave "returns" a probability of 0.95 and "account" only 0.03. With set A, "billing" came first at 0.53, with "account" second at 0.25. With set A reversed, "account" jumped to 0.89. The same message and the same rules, and the right answer went from 3 in 100 to 89 in 100, depending only on which examples came before it and in what order.

For "Please explain the two separate amounts on my statement", a billing question, the three prompts gave three different winners: "returns" at 0.59 with no examples, "billing" at 0.91 with set A, and "account" at 0.47 with set A reversed, with "billing" second at 0.23. For "The courier left the box at the wrong address", "delivery" won only with set A, at 0.77; with no examples and with the reversed set, "returns" won at 0.92 and 0.73.

Two things follow from these numbers. First, examples do not flip a switch inside the model; they move the odds, sometimes a little and sometimes a lot. Second, a big lead under one prompt is no protection against a change to the prompt. Under set A, "billing" led at 0.91 against 0.04 for the statement message, and reversing the order alone made "account" the winner. The margin (how far the winning answer is ahead of the second one) tells you how sure the model is under this exact prompt, not whether the answer will survive a different one.

So there is no shortcut: you cannot tell from one prompt's odds how fragile (easily broken by a small change) it is. Ollama can return these odds with logprobs (the odds, stored on a log scale), and they are useful for seeing how sure the model is, but the only way to find out whether an answer survives a different order or a different set is to run that order or that set.

Set B used four different messages, one per category, in the same order as set A. I wrote set B after seeing set A's results, so it is not a blind test; keep that in mind when you compare the two. Its messages were: "Why was I charged a cancellation fee?", "Can I choose a delivery time slot?", "Is there a fee for sending back a mattress?" and "Please stop sending me newsletters." Llama got 36 of 40 with set B, against 33 with set A. Its mistakes shrank to four, each a different kind.

So with four examples, one per category, llama scored anywhere from 28 to 36 depending on which four messages were used and in what order. That spread of 8 is more than half of the 13-point gain that adding examples gave in the first place. The examples helped, but how much they helped depended on which examples were picked and on their order, which neither the instruction nor the categories say anything about.

On qwen, the three prompts with four examples gave 38 to 39. The same pattern, but small.

A common worry about examples is that the model will copy their answers: show it three billing examples, and it will say "billing" to everything. The last two prompts test that.

With three billing examples, llama answered "billing" only 4 times, fewer than the 10 billing messages in the set. Its old habit came back: it answered "returns" 26 times, and it got 22 of 40 right, barely better than no examples at all.

With three account examples, llama answered "account" only 7 times, and got 34 of 40 right, nearly as good as the best balanced set. I cannot tell from this lab why three account examples worked so much better than three billing examples. One possibility is that the account examples happened to break llama's "returns" habit across all categories (returns answers fell from 30 to 12), while the billing examples did not (26). With one run and three examples each, that stays a guess.

What the numbers do show is that the fear of copying did not come true here, on either model. Qwen scored 37 and 36 with these prompts, close to its 36 with no examples.

Put the two models side by side and the main lesson is clear. On qwen2.5:3b, which already got 36 of 40 without examples, no set of examples changed the score by more than 3. On llama3.2:3b, which got only 20, examples added up to 16 right answers, and choosing and ordering them differently moved the result by up to 8.

In this lab, examples helped the weaker model most, and that is also where their choice and order mattered most. A prompt that works only because of one set of examples in one order that happened to work is fragile: change the examples a little, or change the model, and the result can move a long way.

Four short examples added 65 tokens to llama's prompt: 73 tokens with none, 138 with set A (the median, the middle value of the 40, counted the way Ollama reports them, including the chat template, the hidden text Ollama wraps around your message). On this task, those 65 tokens bought 13 more right answers, a good trade.

Examples are fixed text: they are the same on every call. As lesson 6 of the previous chapter showed, fixed text at the start of a prompt can be reused from one call to the next, as long as nothing that changes comes before it. So put the examples after the rules and before the customer's message, and keep them word for word the same.

This script sends three messages to llama3.2:3b with no examples, with set A, and with set A reversed.

Before you run this lab. It uses llama3.2:3b, running in Ollama on your own computer. If you have not set that up yet, the lab setup guide shows how to install Ollama, download a model and check that everything works, on macOS, Windows or Linux; download this one with ollama pull llama3.2:3b. You can use a different model instead, and your numbers will differ from the ones in this lesson.

"""The same prompt with no examples, four examples, and the same four examples in reverse order.

This one uses llama3.2:3b, where the effect is easy to see. Pull it first (see the lab setup guide):
    ollama pull llama3.2:3b
    python fewshot_demo.py
"""
import json
import urllib.request

RULES = ("Put this customer message into one of these categories: billing, delivery, returns, account.\n\n"
         "Answer with the category word only, in lower case, and nothing else.")
EXAMPLES = [("My receipt shows a different amount from my card statement.", "billing"),
            ("The tracking number you sent does not work.", "delivery"),
            ("Can I send back a kettle that I opened?", "returns"),
            ("I cannot find the settings page to change my name.", "account")]
MESSAGES = ["Please explain the two separate amounts on my statement.",     # billing
            "The courier left the box at the wrong address.",               # delivery
            "Please delete my account and all my data."]                    # account


def ask(text):
    body = {"model": "llama3.2:3b", "stream": False, "messages": [{"role": "user", "content": text}],
            "options": {"temperature": 0, "num_predict": 20}}
    req = urllib.request.Request("http://localhost:11434/api/chat", data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())["message"]["content"]


def prompt(examples, msg):
    shots = [f"Message: {m}\nCategory: {c}" for m, c in examples]
    return "\n\n".join([RULES] + shots + [f"Message: {msg}\nCategory:"])


for msg in MESSAGES:
    none = ask(prompt([], msg))
    four = ask(prompt(EXAMPLES, msg))
    reverse = ask(prompt(EXAMPLES[::-1], msg))
    print(f"\n{msg}")
    print(f"  none: {none.strip()!r:<12} four: {four.strip()!r:<12} reversed: {reverse.strip()!r}")

The lab is scripts/labs/prompting/fewshot.py. It imports lesson 1's rules, test messages and scoring, so every lesson in this chapter measures the same task. It stores one results file per model, and the report mode prints both. A separate odds mode asks for the first token's probabilities on the three demo messages under three of the prompts, which is where the numbers on the odds slide come from, and a choice mode adds set B to an existing results file. In the report, "balanced" is set A, "balanced-rev" is set A reversed, and "balanced-b" is set B. The "usable" column equals "right" in every row, because every reply was a single category word. The columns on the right count how often each category was answered; the "other" column would count replies that named no category, and it stayed at 0: with the format rule in place, both models always answered with one of the four words.

This box has no model. It holds llama's real answers from the lab, one letter per message, and counts the right answers and the most common mistakes for each version of the prompt.

It prints the same numbers as the lab: 20 right with no examples, where the most common mistakes are account answered as returns (8) and billing answered as returns (7); 33 with set A, whose mistakes are all account answered as billing (7); and so on. Try changing most_common(2) to most_common(4) to see every kind of mistake, or count how many times each letter appears in a line to see which answer the model preferred.

The rules. RULES is lesson 1's prompt C: the instruction and the format rule, with no context lines.

The examples. EXAMPLES is set A, four pairs of a message and its category. They are messages that are not in the test set.

Building the prompt. prompt writes each example as "Message: ...", a new line, then "Category: ..." and the answer. It puts a blank line between the rules, each example and the real message. It ends with "Category:", so the model's next word is the answer. EXAMPLES[::-1] is Python for the same list in reverse order.

The call. ask sends the whole prompt as one user message to llama3.2:3b at temperature 0, so it always takes the most likely token and, on one machine, the same prompt gives the same answer. It allows up to 20 tokens, plenty for one word.

Start with no examples, and read the mistakes. Llama's mistakes with no examples pointed straight at the problem: almost everything became "returns". That tells you what the examples need to show.

Add one example per category. A balanced set shows every answer in use. In this lab, a balanced set helped both models; three examples with the same answer did not help reliably.

Try a second set and a second order. This is the step most people skip, and the one this lab says matters most. If a second set or a second order gives a similar score, the prompt is stable. If it moves a lot, as llama's did (28 to 36), the prompt is fragile, and the examples are covering up a weak prompt instead of fixing it. Clearer instructions or context lines, as in lesson 1's prompt D, may be the real fix.

Test on messages the examples never saw. Examples written by looking at your own test messages can make the score look better than it will be on new ones. Keep a set of test messages that you never use for examples.

Use examples when the model gets the task wrong in a way you cannot easily describe. If you can write the fix as a sentence, such as "refunds of money are billing", a context line says it directly, and in lesson 1 context lines took qwen to 40 of 40 (llama was not tested with them). Examples are for patterns that are easier to show than to say.

Use examples when the format is hard to describe. A complicated output shape is often easier to show once than to explain in words.

Do not rely on one set in one order. On the weaker model here, the same four examples in a different order cost 5 right answers. Test at least two sets or two orders before you trust a score.

Weigh the gain against the cost when a prompt already works well. On qwen, set A fixed 3 of the 4 remaining mistakes but added about 65 tokens to every call. Whether that is worth it depends on how costly each mistake is, and every example is more text to keep correct.

Keep examples out of your test set. An example that is also a test message tells you nothing about how the prompt does on new messages.

This lab ran each prompt once, at temperature 0, on two small models. It tried two sets of four examples, two orders of one of them, and two sets of three that share an answer. Set B was written after I had seen set A's results. It did not try more than four examples, many random orders, or a second set of test messages. With only a few versions of each, the numbers show that choice and order can matter a great deal on a weaker model, but not how much on average, or which order is best in general. A proper answer to that would run many random orders and report the spread, which is the kind of test a later lesson in this chapter builds.

Take a prompt of yours that uses examples, or one that could. Score it with no examples on 20 cases and count where the answers go, not only how many are right. Add one example per category, chosen from the kind of case it got wrong. Then score it on a second set of 20 cases that you did not look at while choosing. Swap the order of the examples and score again, and swap in a different set and score again. If the scores are close, keep the examples. If they are far apart, the examples are holding up a prompt that does not really explain the task, and the better fix is usually clearer instructions.