Skip to content
Last Dictionary
Plane
Interpretability & SteeringTraining & Post-trainingLanguage Models & Transformers

ROME

abbreviationAlso: Rank-One Model Editing, rank-one model editing

In plain wordsA method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights.

  1. Rank-One Model Editing: a method for changing a single factual association in a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. language model by a closed-form, rank-one update to the output weight matrix of one middle-layer MLPmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model. module. It treats that matrix as a linear associative memory mapping keys (representations of a subject at its last token) to values (representations that promote facts about it). To make 'The Eiffel Tower is in Rome' true, ROME computes the key for the subject, optimizes a new value vector that makes the model output the new object, and inserts the key–value pair with the minimal rank-one change that leaves other keys' outputs as unchanged as possible. The layer is chosen by causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.. On the authors' CounterFact benchmark ROME generalized to paraphrases better than earlier editors, but it edits one fact at a time, can degrade a model after many sequential edits, and its edits often fail to propagate to related facts; MEMITMEMIT, abbr.: A method for rewriting thousands of facts stored in a language model at once, by spreading small, calculated weight changes across several of its layers. extended it to batches.

    After a ROME edit, the model insists the Eiffel Tower is in Rome, even when the question is reworded.

W^=W+Λ(C−1k∗)⊤,Λ=v∗−Wk∗(C−1k∗)⊤k∗\hat{W} = W + \Lambda \left(C^{-1} k_*\right)^{\top}, \qquad \Lambda = \frac{v_* - W k_*}{\left(C^{-1} k_*\right)^{\top} k_*}

Origin

Introduced by Kevin Meng, David Bau, Alex Andonian and Yonatan Belinkov in 'Locating and Editing Factual Associations in GPT' (February 2022; NeurIPS 2022), together with causal tracing and the CounterFact dataset.

Usage note

In the formula, k* is the subject key, v* the new value, and C the uncentered covariance (second-moment matrix) of keys, estimated in advance from a large text sample.

Related

See also
  • model editingmodel editing, n.: Changing one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing.
  • MEMITMEMIT, abbr.: A method for rewriting thousands of facts stored in a language model at once, by spreading small, calculated weight changes across several of its layers.
  • causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.
  • knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact.
  • parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
  • multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.

Sources

  1. Meng, K., Bau, D., Andonian, A. & Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022. arXiv:2202.05262. arxiv.org/abs/2202.05262

Mentioned in

model editingmodel editing, n.: Changing one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing., causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled., knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact., MEMITMEMIT, abbr.: A method for rewriting thousands of facts stored in a language model at once, by spreading small, calculated weight changes across several of its layers.