History & LandmarksDeep LearningMathematics & Statistics

Hopfield network

nounAlso: Hopfield net, Hopfield model, modern Hopfield network, dense associative memory

In plain wordsA network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint.

A recurrent neural network of binary threshold units with symmetric connections and no self-connections, introduced by John HopfieldJohn Hopfield, prop. n.: An American physicist who showed in 1982 how a network of simple units can store and recall memories, and who shared the 2024 Nobel Prize in Physics for work that enabled machine learning. in 1982 as a model of content-addressable memory. Patterns are stored in the weights, typically by a HebbianHebbian learning, n.: The idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'. rule; units are then updated one at a time, and each update can only lower a global energy function, so the network settles into a local minimum, or attractor. Started from a corrupted or partial pattern, it falls into the nearest stored memory. Capacity is limited to roughly 0.14 patterns per unit before spurious memories dominate. By linking neural computation to the physics of spin glasses, the model helped revive neural-network research in the 1980s. 'Modern' Hopfield networks, or dense associative memories (from 2016), use sharper energy functions to store far more patterns — exponentially many in some versions — and the update rule of the continuous version (2020) coincides with transformer attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones..
Show the Hopfield network half of a stored face and it fills in the rest.

E = -\frac{1}{2} \sum_{i \neq j} w_{ij} \, s_i s_j + \sum_i \theta_i s_i

Origin

Published by the physicist John Hopfield, then at Caltech, as 'Neural Networks and Physical Systems with Emergent Collective Computational Abilities' (PNAS, 1982); a version with continuous-valued units followed in 1984. Related associative-memory models had been studied earlier by Shun'ichi Amari, Kaoru Nakano and William Little.

Usage note

Hopfield shared the 2024 Nobel Prize in Physics with Geoffrey Hinton, in part for this work; see Nobel Prizes for AINobel Prizes for AI, phr.: The two 2024 Nobel Prizes, in physics and chemistry, that honored artificial intelligence research for the first time, recognizing neural-network pioneers and the protein-structure system AlphaFold..

Sources

Hopfield, J. J. (1982). Neural Networks and Physical Systems with Emergent Collective Computational Abilities. Proceedings of the National Academy of Sciences 79(8), 2554–2558.
Ramsauer, H. et al. (2021). Hopfield Networks Is All You Need. ICLR 2021.

Mentioned in

Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them., connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols., Donald HebbDonald Hebb, prop. n.: A Canadian psychologist who proposed in 1949 that connections between brain cells strengthen when the cells are active together, an idea behind many learning rules for neural networks., Hebbian learningHebbian learning, n.: The idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'., John HopfieldJohn Hopfield, prop. n.: An American physicist who showed in 1982 how a network of simple units can store and recall memories, and who shared the 2024 Nobel Prize in Physics for work that enabled machine learning., Nobel Prizes for AINobel Prizes for AI, phr.: The two 2024 Nobel Prizes, in physics and chemistry, that honored artificial intelligence research for the first time, recognizing neural-network pioneers and the protein-structure system AlphaFold., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., associative memoryassociative memory, n.: A memory that is searched by content rather than by location: given part of a stored pattern, or a noisy copy of it, it recalls the whole, much as a few notes can bring back an entire song., energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys., restricted Boltzmann machinerestricted Boltzmann machine, n.: A two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006., Sepp HochreiterSepp Hochreiter, prop. n.: A German computer scientist who explained why early neural networks struggled to learn long sequences and, with Jürgen Schmidhuber, invented the LSTM network that solved the problem., Shun-ichi AmariShun-ichi Amari, prop. n.: A Japanese mathematician who trained multilayer neural networks by gradient descent as early as 1967 and founded information geometry, the source of the 'natural gradient' method.