T5
In plain wordsA 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.
A family of encoder–decoderencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output. transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. (Text-to-Text Transfer Transformer) released by Google in 2019 together with a systematic study of transfer learning for NLP. T5 casts every task as text-to-text: the input is text with a task prefix such as 'summarize:' and the output is text, even for classification and regression, so one model, objective and decoding procedure serve all tasks. It was pretrained on the C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models. web corpus with a span-corruption objective — contiguous spans are replaced by sentinel tokens and the decoder reconstructs them — and released in sizes from 60 million to 11 billion parameters. The paper's ablations of architectures, objectives, data and scale were widely influential. Descendants include multilingual mT5, instruction-tuned Flan-T5 (2022) and T5Gemma (July 2025, adapted from Gemma 2); T5 encoders also serve as text encoders in image generators such as ImagenImagen, prop. n.: Google's family of AI image generators, first shown in 2022, whose research showed that better understanding of language yields pictures that match descriptions more faithfully..
We fine-tuned T5-base to turn support tickets into structured summaries.
Origin
Introduced by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu of Google in 'Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer' (arXiv, October 2019; Journal of Machine Learning Research, 2020).
Related
- encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.
- C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model.
- transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
- instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text.
Sources
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 21. arxiv.org/abs/1910.10683
Mentioned in
BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context., cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt., decoder-only transformerdecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., masked language modelmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example., multi-task learningmulti-task learning, n.: Training one model to do several related jobs at once, such as spotting objects and judging distance in the same photo, so that practice on each job can sharpen the others., Noam ShazeerNoam Shazeer, prop. n.: An American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini., positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model., sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step., AdafactorAdafactor, prop. n.: A memory-saving cousin of the popular Adam training method. Instead of keeping extra statistics for every single weight, it keeps summaries for each row and column of a weight table., byte-level modelbyte-level model, n.: A language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer., C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models., SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed., SuperGLUESuperGLUE, prop. n.: A harder follow-up to the GLUE language-understanding test, released in 2019 after AI systems had already beaten the human baseline on the original.