The Parts of a Prompt: Adding One Piece at a Time

Imagine you give a new helper a pile of customer letters and four boxes, and you say only: "Sort these." They will sort them, but into what? They might make piles by length, by date, by how angry the writer sounds. If you tell them the four boxes are billing, delivery, returns and account, they get much closer. If you also tell them to write only the box name on each letter, you can count the result. And if you explain where the edges are, for example that a refund of money goes in billing, the hard letters stop landing in the wrong box.

A prompt to a language model works the same way. It is not one block of text. It has parts, and each part does one job. This lesson takes a real task, sorting customer messages into four categories, and builds the prompt up one part at a time. After each step it measures two things on 40 messages: how often the model picked the right category, and how often its answer could be used by a program as it stood.

The result is a ladder you can reuse for any prompt: which part to add when the answers come back wrong, and which part to add when they come back right but in a shape your code cannot read.

Here are the five parts this lesson uses. Different books name them differently; the names matter less than the jobs.

Instruction. What you want done, including which answers are allowed. "Put this message into one of these categories: billing, delivery, returns, account."

Format rule. The exact shape of the answer. "Answer with the category word only, in lower case, and nothing else." A format rule is meant to change how the answer is written down, not which category the model picks. As you will see, on this run it did a little of both.

Context. Facts the model needs and cannot guess. Here, one line per category saying what belongs in it, such as "billing: charges, payments, invoices, prices and refunds of money."

Examples. Finished cases: a message and its correct category, written out in full. They show the task instead of describing it.

Input. The text the model works on, here the customer's message, followed by "Category:" so the model knows where its answer goes.

Two more words matter for the scores. An answer is right when the first word of the reply, after any leading "Category:", is the correct category. An answer is usable (the lab prints it as clean) when the whole reply, after making it lower case and dropping spaces at either end, is exactly the category word, so a program can compare it directly. In the lab's code and report, gold means the correct category I wrote down for each message.