residual connection
In plain wordsA shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.
A connection that adds the input of a block of layers to the block's output, y = x + F(x), so the block learns a residual correction F instead of a whole new mapping. Because the identity path carries signals forward and gradients backward unchanged, residual connections make very deep networks trainable, curing the 'degradation problem' in which adding layers to a plain network raised training error. They define ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI. and appear in every transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., where the running sum they create is called the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.; pre-norm designs place normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. inside the residual branch to keep the identity path clean. 'Skip connection' is broader, also covering concatenation of earlier features, as in U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost. and DenseNet. Residual networks can be read as discretized differential equationsneural ordinary differential equation, n.: A neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change. or as ensembles of many shallower paths.
Without the residual connections, the 48-layer model's loss would not go down at all.
Origin
Precursors include shortcut connections across layers in Kevin Lang and Michael Witbrock's 1988 networks and the LSTM's additive cell state (1997). Rupesh Srivastava, Klaus Greff and Jürgen Schmidhuber's highway networks (May 2015) used gated shortcuts to train networks of hundreds of layers; the ungated identity shortcuts of Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun's ResNet (December 2015) made the idea universal.
Usage note
Credit is debated: Jürgen Schmidhuber has argued that ResNets are highway networks with gates held open, while the ResNet authors stressed that their shortcuts are parameter-free identities that never close.
Related
- ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- highway networkhighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable.
- vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
- U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.
- layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.
- neural ordinary differential equationneural ordinary differential equation, n.: A neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change.
Sources
- He, K., Zhang, X., Ren, S. & Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv:1512.03385; CVPR 2016. arxiv.org/abs/1512.03385
- Srivastava, R. K., Greff, K. & Schmidhuber, J. (2015). Highway Networks. arXiv:1505.00387. arxiv.org/abs/1505.00387
Mentioned in
layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., adapteradapter, n.: A small trainable add-on slotted into a large frozen AI model to teach it a new task or style, like a plug adapter that lets one device work in many countries., Jürgen SchmidhuberJürgen Schmidhuber, prop. n.: A German computer scientist who co-invented the LSTM, a neural network with memory that powered much of speech recognition and translation in the 2010s, and who argues that early pioneers go under-credited., Kaiming HeKaiming He, prop. n.: A Chinese computer scientist whose ResNet design, with shortcut connections that let networks grow hundreds of layers deep, is among the most cited research in all of science., layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots., loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys., normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training., residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document., ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI., transformer blocktransformer block, n.: The repeating building unit of a transformer. A model is a stack of dozens of identical blocks, each pairing an attention step that mixes information between words with a step that processes each word., U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., WaveNetWaveNet, prop. n.: A DeepMind model that generates speech and other sounds one tiny audio sample at a time, producing voices far more natural than earlier computer speech., weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., ConformerConformer, prop. n.: A neural-network design for speech recognition that combines a transformer, which relates distant parts of a recording, with convolutions, which pick up fine local sound patterns., DenseNetDenseNet, prop. n.: An image-recognition network in which every layer receives the outputs of all the layers before it, so features are reused rather than relearned., highway networkhighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable., InceptionInception, prop. n.: A family of Google image-recognition networks, the first called GoogLeNet, whose building blocks examine each patch of an image at several scales at once while keeping computation low., MobileNetMobileNet, prop. n.: A family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy., neural ordinary differential equationneural ordinary differential equation, n.: A neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change.