Skip to content
Last Dictionary
Plane
Deep LearningLanguage Models & TransformersVision, Speech & Perception

long short-term memory

nounAlso: LSTM, LSTM network, xLSTM

In plain wordsA type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.

  1. A recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. unit built around a second memory vector, the cell state, which from step to step is changed only by elementwise scaling and addition; along this additive path, originally called the 'constant error carousel', information and error signals can survive hundreds of steps, countering the vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.. Three sigmoidsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on. gates, each computed from the current input and the previous hidden state, regulate the path: the forget gate sets how much of the old cell state is kept, the input gate how much of a new candidate is added, and the output gate how much of the tanh-squashed cell state becomes the new hidden state. From about 2007 to 2017 LSTMs were the workhorse of sequence modeling, behind leading systems for speech and handwriting recognition, machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do. and language modeling. xLSTM (2024) revisited the design with exponential gating and a parallelizable matrix memory to compete with transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it..

    A two-layer LSTM with 512 units was our strongest baseline before we tried transformers.

ct=ft⊙ct−1+it⊙gt,ht=ot⊙tanh⁡(ct)\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \mathbf{g}_t, \qquad \mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)

Origin

Sepp Hochreiter analyzed vanishing gradients in his 1991 diploma thesis supervised by Jürgen Schmidhuber; the two introduced LSTM in a 1995 technical report and the much-cited 1997 Neural Computation paper. Felix Gers, Schmidhuber and Fred Cummins added the now-standard forget gate (1999–2000). Google deployed LSTMs in speech recognition (2015) and Google Translate (2016); a team led by Hochreiter published xLSTM in May 2024.

Related

See also
  • recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
  • gated recurrent unitgated recurrent unit, n.: A simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step.
  • vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
  • backpropagation through timebackpropagation through time, n.: The way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights.
  • sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step.
  • highway networkhighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable.
  • bidirectional recurrent neural networkbidirectional recurrent neural network, n.: A recurrent network that reads a sequence both forwards and backwards, so its understanding of each word or sound can draw on what comes after it as well as what came before.
Contrast with
  • transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.

Sources

  1. Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780.
  2. Gers, F. A., Schmidhuber, J. & Cummins, F. (2000). Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10), 2451–2471.

Mentioned in

architecturearchitecture, n.: The overall design of an AI model, meaning what kinds of parts it has and how they are connected, as opposed to the specific numbers it learns during training., backpropagation through timebackpropagation through time, n.: The way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights., gated recurrent unitgated recurrent unit, n.: A simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step., image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems., Jürgen SchmidhuberJürgen Schmidhuber, prop. n.: A German computer scientist who co-invented the LSTM, a neural network with memory that powered much of speech recognition and translation in the 2010s, and who argues that early pioneers go under-credited., long-term memorylong-term memory, n.: Memory that lasts. For an AI agent, it means information saved outside the conversation — facts, past events, learned procedures — that can be looked up and used again days or months later., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step., sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on., tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., AlphaStarAlphaStar, prop. n.: A DeepMind AI that learned to play the real-time strategy game StarCraft II well enough to reach its highest public rank, Grandmaster, in 2019., bidirectional recurrent neural networkbidirectional recurrent neural network, n.: A recurrent network that reads a sequence both forwards and backwards, so its understanding of each word or sound can draw on what comes after it as well as what came before., connectionist temporal classificationconnectionist temporal classification, n.: A training method that lets a network learn to transcribe speech or handwriting without being told exactly where each letter or sound begins and ends., ELMoELMo, prop. n.: A 2018 model whose word representations changed with context, so 'bank' got different numbers in 'river bank' and 'bank loan'. It was a stepping stone between fixed word vectors and BERT., gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through., highway networkhighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable., neural Turing machineneural Turing machine, n.: A neural network attached to an external memory bank that it learns to read from and write to, an early attempt to give networks something like a computer's working memory., OpenAI FiveOpenAI Five, prop. n.: A team of five AI players built by OpenAI that learned the team video game Dota 2 by playing against itself, and in April 2019 beat the reigning world-champion human team., Sepp HochreiterSepp Hochreiter, prop. n.: A German computer scientist who explained why early neural networks struggled to learn long sequences and, with Jürgen Schmidhuber, invented the LSTM network that solved the problem.