backpropagation
In plain wordsThe method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
An algorithm that works out how sensitive a network's lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. is to each of its weights, the gradient that training needs, by running the chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer. in reverse: it starts from the error at the output and moves back through the layers toward the input, reusing each layer's result for the one before it. A forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. computes and stores each layer's outputs; a backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change. then propagates the derivative of the loss backward, at each layer multiplying the incoming gradient by the layer's local derivative (a vector–Jacobian product) to obtain both the gradients for that layer's weights and the gradient to pass further back. The total cost is a small constant multiple of the forward computation, however many weights there are, which is what makes training networks with billions of parameters feasible; the price is the memory needed to store intermediate values. Backpropagation is reverse-mode automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically. applied to neural networks, and it only computes gradients — an optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. such as stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. performs the update. Applied to unrolled recurrent networks it becomes backpropagation through timebackpropagation through time, n.: The way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights.. Doubts about its biological plausibility motivate alternatives (feedback alignment, target propagation, forward-forward learning), none yet competitive at scale.
Backprop through the frozen encoder wastes memory; detach it so gradients stop at the adapter.
Origin
Credit is disputed. Its mathematical core, reverse-mode differentiation, appeared in Seppo Linnainmaa's 1970 master's thesis; related gradient methods arose in optimal control (Henry Kelley, 1960; Arthur Bryson, 1961; Stuart Dreyfus, 1962). Paul Werbos discussed applying it to neural networks in his 1974 thesis and did so explicitly in 1982; David Parker and Yann LeCun independently described it around 1985. The 1986 Nature paper of David Rumelhart, Geoffrey Hinton and Ronald Williams showed that it learns useful internal representations and popularized it under the name back-propagation.
Usage note
Often used loosely for the whole training procedure; strictly it is only the gradient computation. The 1986 date marks popularization, not invention.
Related
- automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically.
- chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer.
- forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction.
- backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change.
- computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward.
- gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
- vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
Sources
- Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533–536. doi.org/10.1038/323533a0
- Linnainmaa, S. (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics 16, 146–160.
- Schmidhuber, J. (n.d.). Who Invented Backpropagation? Web page, IDSIA. people.idsia.ch/~juergen/who-invented-backpropagation.html
Mentioned in
deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract., Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers., gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., symbolic AIsymbolic AI, n.: An approach to AI that writes knowledge down as human-readable symbols and rules, such as 'every bird has wings', and reasons by manipulating them step by step instead of learning patterns from data., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., Yann LeCunYann LeCun, prop. n.: A French-American computer scientist who pioneered the neural networks that recognize images, led Meta's AI research for over a decade, and argues that chatbots alone will not reach human-level AI., activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., algorithmalgorithm, n.: A precise, step-by-step procedure for solving a problem or doing a task, like a recipe a computer can follow. In the news, 'the algorithm' often means software that ranks what you see online., automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically., backpropagation through timebackpropagation through time, n.: The way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights., backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer., computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward., connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols., credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game., David RumelhartDavid Rumelhart, prop. n.: An American psychologist who, with Geoffrey Hinton and Ronald Williams, showed in 1986 how to train multilayer neural networks, launching the modern revival of the field., derivativederivative, n.: How fast something changes as one of its inputs changes — the steepness of a curve at a point. Training an AI means repeatedly following such slopes downhill., end-to-end learningend-to-end learning, n.: Training one AI system to go straight from raw input, such as camera images, to the final output, such as steering commands, instead of chaining hand-designed steps., exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down., feedforward neural networkfeedforward neural network, n.: A neural network laid out like an assembly line: each stage hands its result only to later stages, never back, so every answer depends on the current input alone and nothing is remembered between inputs., forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., godfathers of AIgodfathers of AI, phr.: A media nickname for three scientists — Geoffrey Hinton, Yoshua Bengio and Yann LeCun — whose decades of work on neural networks underpins modern AI. They shared computing's top prize in 2019., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill.