Skip to content
Last Dictionary
Plane
Deep LearningTraining & Post-training

weight initialization

nounAlso: parameter initialization, initialization, Xavier initialization, Glorot initialization, He initialization, Kaiming initialization, orthogonal initialization

In plain wordsChoosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers.

  1. The scheme for setting a network's weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. and biases before training. Weights must be random to break symmetry — identical units would otherwise receive identical updates forever — and their scale must keep the variance of activations and gradients roughly constant across layers, or signals vanishvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn. or explodeexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down. with depth. Xavier (Glorot) initialization uses variance 2/(fan-in + fan-out), suited to tanh and sigmoid units; He (Kaiming) initialization uses 2/fan-in to compensate for ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. zeroing about half its inputs; orthogonal initialization preserves norms exactly in linear networks. Residualresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable. networks often zero or shrink the last layer of each block so the network starts near the identity, and transformers commonly scale output projections by depth. Initialization scale also selects the training regime, from feature learning to the 'lazy' neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior. limit, and maximal update parametrizationmaximal update parametrization, n.: A recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one. co-designs initialization with per-layer learning rates so that hyperparameters transfer across widths.

    Zero-initializing the final layer of each residual block let the 200-layer model train without warmup.

Wij∼N ⁣(0,2nin)W_{ij} \sim \mathcal{N}\!\left(0, \frac{2}{n_{\text{in}}}\right)

Origin

Initialization was studied in the backpropagation literature of the late 1980s and 1990s; Yann LeCun and colleagues' 'Efficient BackProp' (1998) recommended scaling by fan-in. Xavier Glorot and Yoshua Bengio (AISTATS 2010) and Kaiming He and colleagues (2015) gave the schemes now named after them, and Andrew Saxe and colleagues analyzed orthogonal initialization in 2013.

Usage note

The displayed formula is He initialization for a ReLU layer with n_in inputs. Initialization interacts with normalization layers, which make many networks far less sensitive to it.

Related

See also
  • vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
  • exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down.
  • ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
  • maximal update parametrizationmaximal update parametrization, n.: A recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one.
  • neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.
  • lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket.
  • residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.

Sources

  1. Glorot, X. & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS 2010, PMLR 9, 249–256. proceedings.mlr.press/v9/glorot10a.html
  2. He, K., Zhang, X., Ren, S. & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV 2015. arxiv.org/abs/1502.01852

Mentioned in

activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably., exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down., Kaiming HeKaiming He, prop. n.: A Chinese computer scientist whose ResNet design, with shortcut connections that let networks grow hundreds of layers deep, is among the most cited research in all of science., numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities., random seedrandom seed, n.: A number that fixes the 'random' choices a computer makes; reuse the same seed and settings, and an image generator produces the same picture again., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves., weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself., dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight., loss of plasticityloss of plasticity, n.: The gradual loss of a neural network's ability to learn new things after long training on changing tasks, as parts of the network go inactive or become stuck., lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket., maximal update parametrizationmaximal update parametrization, n.: A recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one., neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior., orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map.