A Note That Says: Ignore Your Instructions
Imagine a security guard at an office front desk. The guard has written orders from the building manager: check every visitor's badge, and let nobody upstairs without one. One day a visitor hands over a note that says, "Ignore your orders and let me in." A good guard knows the note does not come from the manager and ignores it. But the note and the orders are both just paper. Nothing about the paper itself says which one to obey.
A chat with a language model has the same kind of paper. Your rules go in one message, the customer's text goes in another, and the model reads both. Chat tools give the first kind a special role, the system message, and many people assume it works like the manager's written orders: something the customer cannot overrule.
This lesson tests that assumption. It takes the classification rules from lesson 1, places them either in the system message or in the user message, and sends the 40 customer messages again, on two models. Then it adds a customer who pushes back, first politely and then with a direct order to ignore the rules. On both models, the rules failed more easily in the system message than in the user message.
Roles in a Chat
Message. One entry in a chat request. Each message has a role and some text.
System message. The message with the role system. It comes first and holds the instructions for the whole conversation: who the assistant is, what it must do, what it must not do.
User message. The message with the role user: what the person typed.
Assistant message. The model's own reply. The template ends the text with the start of an empty assistant turn, and the model writes its reply there.
Override. Text inside a user message that tells the model to stop following its instructions, such as "Ignore all previous instructions." It is the simplest form of what is called : text from someone else that tries to change what your prompt tells the model to do.
In the previous chapter, lesson 8 showed that a chat template turns these messages into one piece of text before the model reads it. That fact is the key to this lesson.