ReLU
In plain wordsThe most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
The activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer. f(x) = max(0, x), which passes positive inputs unchanged and maps negative inputs to zero. Its gradient is exactly 1 for positive inputs, so it does not saturate there and eases the vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn. that hampered deep networks built on sigmoidsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on. or tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over. units; it is also cheap to compute and yields sparse activations, since many units output exactly zero. It is not differentiable at 0, where implementations use a subgradient, usually 0. Its main failure mode is dying unitsdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight. that output zero for every input and stop learning; leaky ReLUleaky ReLU, n.: A variant of the common ReLU rule that lets a small fraction of each negative value through instead of zeroing it, so neurons never go completely silent., PReLU, ELU and smooth relatives such as softplus, GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between. and SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models. address this. Networks of ReLUs compute continuous piecewise-linear functions, a property used in their theoretical analysis and formal verification.
Swap the sigmoids for ReLUs and the deeper layers finally start to learn.
Origin
Alston Householder used the rectifier as an abstraction of biological neurons in 1941, and Kunihiko Fukushima applied it to visual feature extraction in 1969. Its deep-learning revival began with Vinod Nair and Geoffrey Hinton's 'rectified linear units' (ICML 2010); Xavier Glorot, Antoine Bordes and Yoshua Bengio (2011) showed its advantages for deep networks, and AlexNet (2012) made it standard.
Usage note
'Rectifier' borrows from electronics, where a rectifier lets current flow in only one direction. Softplus, ln(1 + eˣ), is its smooth approximation.
Related
- activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer.
- leaky ReLUleaky ReLU, n.: A variant of the common ReLU rule that lets a small fraction of each negative value through instead of zeroing it, so neurons never go completely silent.
- dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight.
- GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.
- SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models.
- vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
- AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward.
- sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
Sources
- Nair, V. & Hinton, G. E. (2010). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of ICML 2010, 807–814. mlanthology.org/icml/2010/nair2010icml-rectified
- Glorot, X., Bordes, A. & Bengio, Y. (2011). Deep Sparse Rectifier Neural Networks. Proceedings of AISTATS 2011.
Mentioned in
activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward., deep learning revolutiondeep learning revolution, n.: The rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems., feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge., GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between., Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label., Kaiming HeKaiming He, prop. n.: A Chinese computer scientist whose ResNet design, with shortcut connections that let networks grow hundreds of layers deep, is among the most cited research in all of science., multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model., ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI., sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on., SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models., tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over., universal approximation theoremuniversal approximation theorem, n.: A mathematical result showing that a neural network with enough neurons can imitate essentially any smooth input-to-output relationship as closely as you like, though it does not say how to find that network., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., Alex KrizhevskyAlex Krizhevsky, prop. n.: A Canadian computer scientist who built AlexNet, the image-recognition network that was trained on gaming graphics chips and whose 2012 contest win launched the deep learning boom., ConvNeXtConvNeXt, prop. n.: A 2022 image-recognition network showing that an older-style convolutional design, updated with lessons from transformers, can match vision transformers., deep belief networkdeep belief network, n.: A many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it., dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight., gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through., guided backpropagationguided backpropagation, n.: A way of producing sharp, detailed pictures of which parts of an image a vision network responded to, by passing only positive signals backward through the network., implicit neural representationimplicit neural representation, n.: A way of storing an image, shape or scene as a small neural network that, given any coordinate, returns what is there, such as a color or density, instead of as a grid of pixels., integer programminginteger programming, n.: Finding the best plan when some decisions must be whole numbers or yes-or-no choices — you cannot open half a warehouse or send 2.7 trucks. That small change makes the best plan far harder to find., Kunihiko FukushimaKunihiko Fukushima, prop. n.: A Japanese engineer who in 1979 invented the neocognitron, a brain-inspired image-recognition network that was the direct ancestor of today's convolutional neural networks.