SwiGLU
In plain wordsA design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.
A gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through. variant used as the feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge. of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. block. The input x is projected twice; one projection passes through SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models. (Swish) and multiplies the other elementwise, and a third matrix maps the product back to the model width. Because it has three weight matrices instead of two, the hidden size is usually cut by a third (about 8/3 × d_model instead of 4 × d_model) to keep the parameter count equal. In Noam Shazeer's 2020 comparisons it and GEGLU (the GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.-gated version) gave lower perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll. than ReLU or GELU feed-forward layers at equal parameters and compute. It has since become the default feed-forward design in PaLM, LLaMA and most later open-weight language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code..
Like LLaMA, the model uses SwiGLU with an intermediate size of 11,008 for a hidden size of 4,096.
Origin
Introduced by Noam Shazeer in 'GLU Variants Improve Transformer' (arXiv, February 2020), which built on the gated linear units of Yann Dauphin and colleagues (2016). The paper offered no explanation of why the variants work, attributing their success 'as all else, to divine benevolence'. Google's PaLM (2022) and Meta's LLaMA (2023) carried it into large language models.
Related
- gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through.
- SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models.
- GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.
- feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.
Sources
- Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202. arxiv.org/abs/2002.05202
Mentioned in
transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge., GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between., multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model., Noam ShazeerNoam Shazeer, prop. n.: An American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini., SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models., gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through.