instruction hierarchy
In plain wordsA rule built into AI assistants about whose instructions win when they conflict: the developer's beat the user's, and the user's beat any instructions hidden in a web page or document.
An ordering of trust among the sources of text in a model's context — typically the model provider's policy (model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict.), then system or developersystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply. messages, then user messages, then tooltool, n.: Any outside capability an AI model is allowed to use — a web search, a calculator, a code runner, a database or an app — described to the model so it can ask for it by name. outputs and retrieved content — together with training that teaches the model to follow higher-privileged instructions and to ignore or refuse conflicting lower-privileged ones. Lower-level instructions consistent with higher ones (a user asking for a different format) are followed; conflicting ones (a web page telling an agent to send out the user's files) are not. It is a primary defense against prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., indirect prompt injectionindirect prompt injection, n.: A prompt-injection attack in which the trick instructions are hidden in content the AI reads on its own — a web page, an email, a file — rather than typed by the person using it. and system-prompt extraction, though not a complete one, and it governs how deployed assistants arbitrate between operators and users.
Thanks to the instruction hierarchy, the agent ignored the commands hidden in the PDF it was summarizing.
Origin
Formalized by Eric Wallace and colleagues at OpenAI in 'The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions' (April 2024), which argued that treating system prompts and untrusted text as equals underlies many attacks. OpenAI's Model Spec (first published May 2024) describes a related 'chain of command'.
Related
- system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply.
- prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger.
- indirect prompt injectionindirect prompt injection, n.: A prompt-injection attack in which the trick instructions are hidden in content the AI reads on its own — a web page, an email, a file — rather than typed by the person using it.
- model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict.
- instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related.
- guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data.
Sources
- Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J. & Beutel, A. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv:2404.13208. arxiv.org/abs/2404.13208
Mentioned in
prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., indirect prompt injectionindirect prompt injection, n.: A prompt-injection attack in which the trick instructions are hidden in content the AI reads on its own — a web page, an email, a file — rather than typed by the person using it., instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related., model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict., special tokenspecial token, n.: A reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes., steerabilitysteerability, n.: How easily an AI model's behavior — its tone, style, persona or rules — can be adjusted by the people using or deploying it, for example through instructions, without retraining., system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply.