steering vector
In plain wordsA direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.
A vector added to a model's activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., usually in the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. at one or more layers, during the forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. to shift its outputs toward a target property. A common recipe takes the difference between mean activations on contrasting inputs (for example, honest versus dishonest completions) and adds a scaled multiple of it at inference time; vectors can also be found by optimization or taken from sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. features. Steering is cheap and reversible compared with fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., and its success supports the linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.. Effects can be brittle, degrade fluency at high strength and fail to generalize across prompts.
Adding the honesty steering vector at layer 14 cut false answers but made the model wordier.
Origin
Nishant Subramani, Nivedita Suresh and Matthew Peters (2022) extracted 'latent steering vectors' by optimization to control pretrained language models. Alexander Matt Turner and colleagues (2023) showed that simply adding activation differences works ('activation addition', ActAdd); contrastive activation addition (Rimsky et al., 2023) and representation engineering (Zou et al., 2023) followed.
Related
- activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model.
- activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior.
- activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes.
- contrastive activation additioncontrastive activation addition, n.: A way to make a steering signal more reliable by averaging the difference in a model's internal state across hundreds of example answers that do and don't show a behavior, such as flattery.
- representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly.
- persona vectorpersona vector, n.: A direction inside a language model's internal activity linked to a character trait, such as being flattering, malicious or prone to making things up, used to watch for or control that trait.
- linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- prompt engineeringprompt engineering, n.: The skill of writing requests to an AI — picking the words, examples, format and background information it receives — so that it reliably gives the result you want, without retraining it.
Sources
- Subramani, N., Suresh, N. & Peters, M. (2022). Extracting Latent Steering Vectors from Pretrained Language Models. Findings of ACL 2022. aclanthology.org/2022.findings-acl.48
- Turner, A. M. et al. (2023). Steering Language Models With Activation Engineering (first circulated as 'Activation Addition: Steering Language Models Without Optimization'). arXiv:2308.10248. arxiv.org/abs/2308.10248v4
Mentioned in
activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior., introspectionintrospection, n.: Looking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up., linear algebralinear algebra, n.: The branch of mathematics about lists and grids of numbers and the even, straight-line transformations between them. It is the everyday language in which neural networks are written and computed., linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly., steerabilitysteerability, n.: How easily an AI model's behavior — its tone, style, persona or rules — can be adjusted by the people using or deploying it, for example through instructions, without retraining., steeringsteering, n.: Guiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it., The Last DictionaryThe Last Dictionary, prop. n.: This book: a dictionary of intelligence, human and otherwise, which explains each word first in plain language, then precisely, and says when it was born., TruthfulQATruthfulQA, prop. n.: A test of whether AI models repeat popular myths and misconceptions — questions to which many people, and much of the internet, give a confidently wrong answer., activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes., activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., assistant axisassistant axis, n.: The main direction in a chatbot's internal activity that measures how much it is acting as its usual helpful-assistant self, as opposed to some other character it has slipped into., concept injectionconcept injection, n.: An experiment that plants a specific concept directly into an AI model's internal activity, then asks the model whether it notices anything unusual, to test whether its reports about itself are genuine., contrastive activation additioncontrastive activation addition, n.: A way to make a steering signal more reliable by averaging the difference in a model's internal state across hundreds of example answers that do and don't show a behavior, such as flattery., control vectorcontrol vector, n.: A steering signal saved as a small file that can be loaded alongside an open AI model to turn a trait — say, cheerfulness or laziness — up or down while it generates text., directional ablationdirectional ablation, n.: Erasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does., feature steeringfeature steering, n.: Steering an AI model by turning up or down specific concepts that interpretability tools have found inside it — for example, making a model obsessed with a bridge by maxing out its 'bridge' concept., function vectorfunction vector, n.: A compact internal signal, found inside a language model, that stands for a whole task — such as 'translate into French' or 'give the opposite word' — and can trigger that task when added to the model., in-context vectorin-context vector, n.: A steering signal distilled from a few worked examples shown to a language model, so the model can follow the examples' pattern without having them repeated in every prompt., linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction., persona vectorpersona vector, n.: A direction inside a language model's internal activity linked to a character trait, such as being flattering, malicious or prone to making things up, used to watch for or control that trait., plug and play language modelplug and play language model, n.: An early way to steer what a language model writes — its topic or mood — by letting a small add-on classifier nudge the model's internal state at each word, without retraining the model., refusal directionrefusal direction, n.: A single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests., representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly.