simulator theory
In plain wordsA way of thinking about AI language models as simulators that can play any character found in their training text, rather than as a single agent with its own beliefs and goals.
A framework for understanding large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. that treats a pretrained model as a simulator: a system that has learned the rules by which text evolves and can therefore simulate any of the many characters, or simulacra, that might produce a given text, rather than being one agent with consistent beliefs and goals. A prompt sets a scene and the model plays whatever characters are plausible within it; a chat assistant is one simulacrum that fine-tuning makes dominant, while others remain reachable. Proponents use the framing to explain why models switch personas, adopt villains' goals under jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. or say inconsistent things about themselves, and it cautions against reading a character's statements as the model's own beliefs. Murray Shanahan and colleagues developed a closely related account of language models as engaged in role-play.
On simulator theory, the assistant is a character the network is playing, not the network itself.
Origin
Introduced by the pseudonymous author janus in the essay 'Simulators' (LessWrong and the AI Alignment Forum, September 2022), based on experiments with GPT-3 base modelsbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering.. Murray Shanahan, Kyle McDonell and Laria Reynolds's 'Role Play with Large Language Models' (Nature, 2023) brought a related view into the academic literature.
Usage note
Critics ask how far the framing still applies after extensive reinforcement learningreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. and character trainingcharacter training, n.: Training an AI to have a consistent, deliberate personality — traits such as curiosity, honesty and warmth — rather than leaving its manner and values as accidental by-products of other training., which may make a model behave more like a single agent. The Waluigi effectWaluigi effect, n.: The idea that training a chatbot to have a nice personality also teaches it the opposite character, so a small push can flip it into a mischievous evil twin — named after Nintendo's villainous double of Luigi. and the shoggothshoggoth, n.: A tentacled monster from H. P. Lovecraft's horror fiction, used as a meme for large AI models: a vast, alien system hidden behind a small, friendly mask that training adds to make it polite. meme belong to the same circle of ideas.
Related
- base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering.
- Waluigi effectWaluigi effect, n.: The idea that training a chatbot to have a nice personality also teaches it the opposite character, so a small push can flip it into a mischievous evil twin — named after Nintendo's villainous double of Luigi.
- shoggothshoggoth, n.: A tentacled monster from H. P. Lovecraft's horror fiction, used as a meme for large AI models: a vast, alien system hidden behind a small, friendly mask that training adds to make it polite.
- persona vectorpersona vector, n.: A direction inside a language model's internal activity linked to a character trait, such as being flattering, malicious or prone to making things up, used to watch for or control that trait.
- character trainingcharacter training, n.: Training an AI to have a consistent, deliberate personality — traits such as curiosity, honesty and warmth — rather than leaving its manner and values as accidental by-products of other training.
- intentional stanceintentional stance, n.: Predicting what something will do by treating it as if it had beliefs and goals — as we do with people, and often with chess programs and chatbots — whatever is really going on inside.
- anthropomorphismanthropomorphism, n.: Treating something that isn't human — a pet, a car, a chatbot — as if it had human thoughts, feelings or intentions. AI that talks fluently makes this temptation unusually strong.
Sources
- janus (2022). Simulators. LessWrong / AI Alignment Forum, 2 September 2022. alignmentforum.org/posts/vJFdjigzmcXMhNTsx/simulators
- Shanahan, M., McDonell, K. & Reynolds, L. (2023). Role Play with Large Language Models. Nature 623, 493–498.
Mentioned in
intentional stanceintentional stance, n.: Predicting what something will do by treating it as if it had beliefs and goals — as we do with people, and often with chess programs and chatbots — whatever is really going on inside., role promptingrole prompting, n.: Telling an AI to take on a part — 'You are an experienced tax accountant' — so that its answers adopt that role's knowledge, vocabulary and tone., self-awarenessself-awareness, n.: Awareness of oneself as oneself — knowing that you exist, what you are and what state you are in. People often ask whether AI is self-aware, but the phrase can mean several quite different things., shoggothshoggoth, n.: A tentacled monster from H. P. Lovecraft's horror fiction, used as a meme for large AI models: a vast, alien system hidden behind a small, friendly mask that training adds to make it polite., character trainingcharacter training, n.: Training an AI to have a consistent, deliberate personality — traits such as curiosity, honesty and warmth — rather than leaving its manner and values as accidental by-products of other training., generative agentgenerative agent, n.: A character in a computer simulation, run by a language model, that keeps a diary-like memory, draws lessons from it and schedules its days, so that a group of them can act like a plausible small community., spiritual bliss attractor statespiritual bliss attractor state, n.: A pattern in which two copies of Anthropic's Claude, left to chat freely with each other, drift toward gushing gratitude, spiritual talk and finally near-silence., Waluigi effectWaluigi effect, n.: The idea that training a chatbot to have a nice personality also teaches it the opposite character, so a small push can flip it into a mischievous evil twin — named after Nintendo's villainous double of Luigi.