Skip to content
Last Dictionary
Plane
Language Models & TransformersDeep Learning

ALiBi

abbreviationAlso: Attention with Linear Biases

In plain wordsA simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on.

  1. Short for Attention with Linear Biases: a way of conveying word orderpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model. without any position vectors, in which each attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns. lowers every query–key score in proportion to how far apart the two positions are, at a fixed head-specific rate drawn from a geometric sequence (for 8 heads: 1/2, 1/4, …, 1/256). Because nearer tokens are always favored, models trained on short sequences stay stable on longer ones: in the authors' experiments a 1.3-billion-parameter model trained on 1,024-token inputs, evaluated on 2,048 tokens, equaled the perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll. of a sinusoidal-position model trained at that length, while training 11% faster with 11% less memory. It was adopted by BLOOM and MPT, but RoPErotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are. plus context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples. methods later prevailed, partly because ALiBi's strong locality can hamper use of distant context.

    Because the model uses ALiBi, we can run it at twice its training length without fine-tuning.

softmax⁡ ⁣(qiK⊤+m⋅[−(i−1),…,−2,−1,0])\operatorname{softmax}\!\left(q_i K^{\top} + m \cdot [-(i-1), \dots, -2, -1, 0]\right)

Origin

Introduced by Ofir Press, Noah A. Smith and Mike Lewis of the University of Washington and Facebook AI Research in 'Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation' (arXiv, August 2021; ICLR 2022).

Usage note

'Extrapolation' here means stable perplexity on longer inputs, not necessarily the ability to retrieve information from far back.

Related

See also
  • positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.
  • rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are.
  • context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples.
  • long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
  • attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later.
  • attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.

Sources

  1. Press, O., Smith, N. A. & Lewis, M. (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv:2108.12409; ICLR 2022. arxiv.org/abs/2108.12409

Mentioned in

attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later., positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model., context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples., rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are.