gated linear unit
In plain wordsA layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through.
A layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage. that projects its input twice and lets one projection, squashed by a sigmoidsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on. into values between 0 and 1, scale the other element by element. The squashed branch works as a learned, input-dependent gate, and because the ungated branch is linear, gradients pass through it without saturating. First used in gated convolutional language models, GLUs are now best known through variants that swap the sigmoid for another function — SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together. (SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models.), GEGLU (GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.) and ReGLU (ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.) — or drop it altogether, leaving a 'bilinear' layer. These variants are the standard feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge. in modern language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.. Gating by elementwise multiplication also underlies LSTMslong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., GRUsgated recurrent unit, n.: A simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step., highway networkshighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable. and the gated activations of WaveNetWaveNet, prop. n.: A DeepMind model that generates speech and other sounds one tiny audio sample at a time, producing voices far more natural than earlier computer speech..
Replacing the ReLU feed-forward block with a gated linear unit lowered perplexity at equal parameter count.
Origin
Introduced by Yann Dauphin, Angela Fan, Michael Auli and David Grangier of Facebook AI Research in 'Language Modeling with Gated Convolutional Networks' (arXiv, December 2016; ICML 2017) as a simpler alternative to the tanh-and-sigmoid gating of Aäron van den Oord and colleagues' 2016 image models. Noam Shazeer's 2020 variants carried GLUs into transformers.
Related
- SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.
- SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models.
- feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge.
- long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.
- highway networkhighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable.
- sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
Sources
- Dauphin, Y. N., Fan, A., Auli, M. & Grangier, D. (2016). Language Modeling with Gated Convolutional Networks. arXiv:1612.08083; ICML 2017. arxiv.org/abs/1612.08083
- Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202. arxiv.org/abs/2002.05202
Mentioned in
GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between., sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on., SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models., highway networkhighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable., squeeze-and-excitation networksqueeze-and-excitation network, n.: An image-network design whose add-on blocks let the network decide, for each picture, which of its internal feature channels to turn up and which to turn down., SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.