sequence-to-sequence model
In plain wordsA model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step.
A model that maps an input sequence to an output sequence of possibly different length, typically with an encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.: an encoder reads the input into a representation, and an autoregressiveautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. decoder generates the output one tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. at a time, conditioned on that representation and its own previous outputs. The first neural versions used LSTMslong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences. or GRUsgated recurrent unit, n.: A simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step. that compressed the input into a single fixed-length vector; attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. (Dzmitry Bahdanau and colleagues, 2014) removed this bottleneck by letting the decoder consult all encoder states, and the transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. (2017) replaced recurrence altogether. Applications include machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do., text summarizationtext summarization, n.: Using a computer to shorten a document, meeting or set of articles into a brief version that keeps the main points. and speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.. Training usually uses teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one.; outputs are produced with greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text. or beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility..
We framed SQL generation as a seq2seq problem: question in, query out.
Origin
Popularized by Ilya Sutskever, Oriol Vinyals and Quoc Le's 'Sequence to Sequence Learning with Neural Networks' (arXiv, September 2014; NIPS 2014), whose deep LSTMs reached 34.8 BLEU on WMT'14 English–French. Kyunghyun Cho and colleagues' RNN encoder–decoder (June 2014) and Nal Kalchbrenner and Phil Blunsom's 2013 translation models were close predecessors; Google Translate switched to neural sequence-to-sequence translation in 2016.
Usage note
Tomáš Mikolov has said he proposed a similar approach earlier without credit. Today 'seq2seq' often denotes encoder–decoder transformers such as T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design., in contrast to decoder-only models.
Related
- encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.
- long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do.
- teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one.
- beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility.
- T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.
Sources
- Sutskever, I., Vinyals, O. & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NIPS 2014; arXiv:1409.3215. arxiv.org/abs/1409.3215
- Bahdanau, D., Cho, K. & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473; ICLR 2015. arxiv.org/abs/1409.0473
Mentioned in
retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., end-to-end learningend-to-end learning, n.: Training one AI system to go straight from raw input, such as camera images, to the final output, such as steering commands, instead of chaining hand-designed steps., gated recurrent unitgated recurrent unit, n.: A simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step., Google BrainGoogle Brain, prop. n.: Google's deep-learning research team from 2011 to 2023, where the transformer — the design behind today's chatbots — and the TensorFlow software were created., Ilya SutskeverIlya Sutskever, prop. n.: A Russian-born Israeli-Canadian computer scientist who co-built the AlexNet image recognizer, helped found OpenAI and lead its research until 2024, and now heads the startup Safe Superintelligence Inc., long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do., natural language generationnatural language generation, n.: Getting computers to produce human language — from turning a spreadsheet into a weather report to writing an essay or a chatbot's reply., NeurIPSNeurIPS, prop. n.: The largest yearly scientific conference on artificial intelligence and machine learning, held each December, where many of the field's landmark papers were first presented., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., special tokenspecial token, n.: A reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes., teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one., text summarizationtext summarization, n.: Using a computer to shorten a document, meeting or set of articles into a brief version that keeps the main points., connectionist temporal classificationconnectionist temporal classification, n.: A training method that lets a network learn to transcribe speech or handwriting without being told exactly where each letter or sound begins and ends., Dzmitry BahdanauDzmitry Bahdanau, prop. n.: A Belarusian computer scientist who, as a student in Yoshua Bengio's lab in 2014, was lead author of the paper that introduced attention, the mechanism later at the core of the transformer., Kyunghyun ChoKyunghyun Cho, prop. n.: A South Korean computer scientist who co-invented the encoder–decoder approach to neural machine translation and the gated recurrent unit, and co-wrote the paper that introduced attention., Oriol VinyalsOriol Vinyals, prop. n.: A Spanish computer scientist who co-created key neural methods for translation and image captioning, led DeepMind's StarCraft-playing AlphaStar, co-led Google's Gemini models, and in 2026 co-founded the startup Discovery Loop., semantic parsingsemantic parsing, n.: Translating a sentence into a precise, machine-executable form — such as a database query or a logical formula — so that a computer can act on what was asked.