Hebbian learning
In plain wordsThe idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'.
A learning principle that strengthens the connection from one unit to another each time activity in the first helps drive activity in the second, so that units which are often active together become more tightly coupled. In its simplest artificial form the weight change is proportional to the product of input and output activity. Pure Hebbian updates are local and unsupervised but unstable, because weights can grow without bound; variants add decay or normalization, as in Oja's rule (1982), which makes a single linear neuron extract the first principal componentprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores. of its inputs, or allow weakening when activity is anti-correlated. Hebbian storage underlies the Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint. and other associative memories, and spike-timing-dependent plasticity, in which the order of firing matters, is a biological refinement of the idea.
The Hopfield network stores its patterns with a one-shot Hebbian rule rather than gradient descent.
Origin
Proposed by the Canadian psychologist Donald O. Hebb in The Organization of Behavior (1949) as the mechanism by which 'cell assemblies' form. The slogan about cells that fire together wiring together came much later and is usually credited to the neuroscientist Carla Shatz (1992); Siegrid Löwel and Wolf Singer used a similar phrase the same year.
Usage note
Unlike backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction., Hebbian rules use only locally available activity and no error signal, which is why they remain of interest for biologically plausible learning and neuromorphic computingneuromorphic computing, n.: Computer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy..
Related
- Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint.
- Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.
- connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols.
- unsupervised learningunsupervised learning, n.: Machine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary.
- neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain.
- McCulloch–Pitts neuronMcCulloch–Pitts neuron, n.: The first mathematical model of a brain cell as a simple on–off switch: it fires if enough of its inputs are on and no blocking input is. Networks of them can carry out logic.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
Sources
- Hebb, D. O. (1949). The Organization of Behavior: A Neuropsychological Theory. New York: Wiley.
Mentioned in
connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols., Donald HebbDonald Hebb, prop. n.: A Canadian psychologist who proposed in 1949 that connections between brain cells strengthen when the cells are active together, an idea behind many learning rules for neural networks., Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint., learninglearning, n.: Getting better at something through experience. For a machine, it means changing its internal settings based on data so that it performs a task better than before., McCulloch–Pitts neuronMcCulloch–Pitts neuron, n.: The first mathematical model of a brain cell as a simple on–off switch: it fires if enough of its inputs are on and no blocking input is. Networks of them can carry out logic., associative memoryassociative memory, n.: A memory that is searched by content rather than by location: given part of a stored pattern, or a noisy copy of it, it recalls the whole, much as a few notes can bring back an entire song., delta ruledelta rule, n.: A simple learning rule for one artificial neuron: adjust each connection in proportion to the error, the gap between the wanted and the actual output, times that connection's input., spiking neural networkspiking neural network, n.: A neural network that, like the brain, communicates with brief pulses, or spikes, sent only when a neuron's charge crosses a threshold. On special chips this can save a great deal of energy.