RWKV
In plain wordsAn open-source language model design that works like an old-style recurrent network when writing text, using constant memory, yet can be trained quickly in parallel like a transformer.
A neural architecture (Receptance Weighted Key Value) designed to train like a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., with all positions processed in parallel, yet run like a recurrent networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., keeping only a fixed-size state while generating. Each block has a time-mixing sublayer — a form of linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words. in which past keys and values are accumulated with learned exponential decay and gated by a 'receptance' vector — and a channel-mixing sublayer that plays the role of the feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge.. During generation each layer just updates its state, so cost per token does not grow with context. Its 2023 paper reported models of up to 14 billion parameters, larger than any dense RNN trained before, that performed about as well as transformers of similar size. Later versions added data-dependent decay (RWKV-6, 'Finch') and generalized delta-rule state updates (RWKV-7, 'Goose', 2025). Like other fixed-state models, it is weaker than attention at exact recall over long contexts.
On the phone, the RWKV model's memory use stays flat however long the conversation gets.
Origin
Developed by Bo Peng as an open-source project (first public code, RWKV-1, in August 2021), later supported by EleutherAI and a volunteer community and now a Linux Foundation AI project, and described in 'RWKV: Reinventing RNNs for the Transformer Era' (Bo Peng and colleagues, May 2023; Findings of EMNLP 2023).
Related
- linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words.
- recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
- state space modelstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.
- MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly.
- hybrid architecturehybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
Sources
- Peng, B., Alcaide, E., Anthony, Q., Albalak, A. et al. (2023). RWKV: Reinventing RNNs for the Transformer Era. Findings of EMNLP 2023. arxiv.org/abs/2305.13048
Mentioned in
recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words., MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly., state space modelstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.