Vague, Soft and Specific: Asking for Exactly What You Want

Suppose you ask a carpenter for "a short shelf". You have a gap of 60 centimetres in mind. The carpenter has a different idea of short, and delivers a shelf of 90 centimetres. For a shelf, that is short, but it does not fit your 60-centimetre gap. Nobody did anything wrong. You had a requirement, a number, and you did not say it. The word "short" felt like it said it, but it only gave a direction.

Instructions to a language model fail in the same way, and they fail more often than people expect, because the model's replies look reasonable. A summary that is five sentences long looks like a summary. A list of fifteen names looks helpful. The failure only shows when your program, your page layout or your reader needed three names, or one sentence.

The previous lesson showed that a prompt has parts. This lesson looks closely at the most important part, the instruction, and at one question: how exactly must you say what you want? It measures five everyday tasks, each asked three ways, with the same check applied to every reply.

Requirement. What the answer must be for you to be able to use it. "One sentence of at most 25 words" is a requirement. So is "exactly three names, one per line".

Vague wording. The request with no requirement at all, the way people often type it: "Summarise this."

Soft word. A word that points in the direction of a requirement without stating it: briefly, a few, short, just. "Summarise this briefly" is a vague request with a soft word added. In this lab I call the middle wording of every task the soft-word wording, though for the verdict task it has no soft word: it names the two answers but not the shape.

Specific wording. The request with the requirement stated in full, as a number or an exact shape: "Summarise this in one sentence of at most 25 words."

Check. A few lines of code that look at a reply and say whether it met the requirement. The main idea of this lesson is that a requirement is specific enough exactly when you could write its check.