ADALINE
In plain wordsAn early learning machine from 1960 that adjusted its settings to shrink its errors step by step, the same basic idea used to train today's neural networks.
A single-layer adaptive unit devised by Bernard Widrow and Marcian (Ted) Hoff at Stanford University in 1960. It computes a weighted sum of its inputs and thresholds it to give a ±1 output, but, unlike the perceptronperceptron, n.: One of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong., it is trained on the linear sum before thresholding: the least-mean-squares (LMS) or Widrow–Hoff rule changes each weight in proportion to the error times the corresponding input. This amounts to stochastic gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. on squared error and converges to a best linear fit even when the classes cannot be separated. Widrow's group built it in hardware using 'memistors', electrochemical variable resistors, and combined several units into MADALINE (Many ADALINEs) in 1962. The LMS algorithm became a foundation of adaptive signal processing, used in modem equalizers, echo cancellers and noise-cancelling systems.
ADALINE's LMS update is the delta rule that backpropagation later generalized to hidden layers.
Origin
Introduced in Widrow and Hoff's paper 'Adaptive Switching Circuits' at the 1960 IRE WESCON convention. The acronym first expanded as ADAptive LINear NEuron; the last word later became Element.
Usage note
Hoff went on to co-invent the Intel 4004 microprocessor.
Related
- perceptronperceptron, n.: One of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong.
- stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
- gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
- Mark I PerceptronMark I Perceptron, prop. n.: A room-sized machine built around 1960 that learned to recognize simple shapes and letters through a camera made of 400 light sensors. It was one of the first machines built to learn from examples.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
- neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain.
- perceptronperceptron, n.: One of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong.
Sources
- Widrow, B. & Hoff, M. E. (1960). Adaptive Switching Circuits. IRE WESCON Convention Record, Part 4, 96–104.
Mentioned in
Bernard WidrowBernard Widrow, prop. n.: An American electrical engineer who in 1960 built ADALINE, an early trainable neural unit, and invented a learning rule that adjusts weights to reduce errors, still at the heart of adaptive filters., delta ruledelta rule, n.: A simple learning rule for one artificial neuron: adjust each connection in proportion to the error, the gap between the wanted and the actual output, times that connection's input., Mark I PerceptronMark I Perceptron, prop. n.: A room-sized machine built around 1960 that learned to recognize simple shapes and letters through a camera made of 400 light sensors. It was one of the first machines built to learn from examples.