knowledge neuron
In plain wordsAn internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact.
A neuron in the feed-forward (MLPmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.) layers of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. identified as contributing strongly to the model's expression of a particular relational fact. Damai Dai and colleagues found such neurons in BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. by scoring MLP activations with an integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.-style attribution on cloze prompts for a fact and keeping the neurons that scored highly across paraphrases; turning those neurons down made the right answer less likely, turning them up made it more likely, and changing them could revise or delete the fact. The work built on Mor Geva and colleagues' view of feed-forward layers as key–value memories. The broader 'knowledge neuron thesis' — that facts are stored in a few identifiable neurons — has been criticized: the same localization and editing methods work just as well on syntactic, non-factual phenomena, effects are modest, and factual recall appears distributed, so the label is now used with caution.
Suppressing the two knowledge neurons cut the probability of 'Paris' by half, but did not erase the fact.
Origin
Introduced by Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang and Furu Wei in 'Knowledge Neurons in Pretrained Transformers' (arXiv, April 2021; ACL 2022). Jingcheng Niu and colleagues' 'What does the Knowledge Neuron Thesis Have to do with Knowledge?' (ICLR 2024) named and challenged the broader thesis.
Related
- model editingmodel editing, n.: Changing one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing.
- ROMEROME, abbr.: A method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights.
- causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.
- parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
- polysemanticitypolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
Sources
- Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B. & Wei, F. (2022). Knowledge Neurons in Pretrained Transformers. ACL 2022. arXiv:2104.08696. arxiv.org/abs/2104.08696
- Geva, M., Schuster, R., Berant, J. & Levy, O. (2021). Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP 2021.
- Niu, J., Liu, A., Zhu, Z. & Penn, G. (2024). What does the Knowledge Neuron Thesis Have to do with Knowledge? ICLR 2024. arXiv:2405.02421. arxiv.org/abs/2405.02421
Mentioned in
model editingmodel editing, n.: Changing one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing., causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled., ROMEROME, abbr.: A method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights.