Skip to content
Last Dictionary
Plane
Language Models & TransformersDeep Learning

rotary positional embedding

nounAlso: RoPE, rotary position embedding, rotary embedding, RoFormer

In plain wordsA way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are.

  1. A positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model. that rotates query and key vectors by position-dependent angles instead of adding position vectors to embeddings. Each pair of dimensions is treated as a 2-D plane and rotated by m·θ_i at position m, with frequencies θ_i = b^(−2i/d) spaced geometrically (base b = 10,000 originally). Because rotations preserve lengths and the dot product of two rotated vectors depends only on the difference of their angles, the attention score between positions m and n depends on the offset n − m: absolute encoding that yields relative behavior. RoPE adds no parameters and works with KV cachingKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.. It is used by most open-weight LLMslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. released since 2023, including the Llama, Qwen, Mistral and DeepSeek families, and its frequencies are what context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples. methods such as position interpolation, NTK-aware scaling and YaRN rescale.

    We raised the RoPE base to 500,000 before long-context training.

(RΘ,mWqxm)⊤(RΘ,nWkxn)=xm⊤Wq⊤RΘ, n−m Wk xn\left(R_{\Theta,m} W_q x_m\right)^{\top}\left(R_{\Theta,n} W_k x_n\right) = x_m^{\top} W_q^{\top} R_{\Theta,\,n-m}\, W_k\, x_n

Origin

Proposed by Jianlin Su and colleagues at Zhuiyi Technology in Shenzhen, first in Chinese-language blog posts (early 2021) and then in the RoFormer paper (arXiv, April 2021). EleutherAI's GPT-J and GPT-NeoX and Google's PaLM adopted it in 2021–2022, and Meta's LLaMA (2023) made it the de facto standard.

Usage note

Despite the name, nothing is added to the token embeddings; the rotation acts inside attention. Long-context models often raise the base frequency (Llama 3 used 500,000) to slow the rotation.

Related

See also
  • positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.
  • context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples.
  • ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on.
  • self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
  • query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen.
  • long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.

Sources

  1. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B. & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864. arxiv.org/abs/2104.09864

Mentioned in

transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model., ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on., context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples., Fourier transformFourier transform, n.: A way of breaking a signal — a sound, an image, any pattern over time or space — into the simple waves of different frequencies that add up to it, much as a prism splits white light into colors., multi-head latent attentionmulti-head latent attention, n.: An attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory.