Noam Shazeer
In plain wordsAn American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini.
American computer scientist who joined Google in 2000, working on search spelling correction and advertising systems before turning to neural networks. The author-contribution note of 'Attention Is All You NeedAttention Is All You Need, prop. n.: The 2017 Google research paper that introduced the transformer, the neural-network design behind ChatGPT, Claude, Gemini and nearly every other modern AI language system.' (2017) credits him with proposing three core pieces of the original transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.: scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information., multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results. and the parameter-free encoding of token positions. He was lead author of the sparsely gated mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. layer (2017), introduced multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory. (2019) and SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.-style gated feed-forward layers (2020), and co-authored the Adafactor optimizer, Mesh-TensorFlow, T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design. and the Switch Transformer, techniques now common in large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.. He left Google in 2021 to co-found the chatbot company Character.AICharacter.AI, prop. n.: A popular app where people chat with AI characters — invented personas, fictional figures or imitations of real people — that other users have created. with Daniel De Freitas. In 2024 he returned to Google under a licensing agreement with Character.AI and became a technical co-lead of GeminiGemini, prop. n.: Google's family of AI models and its chatbot assistant, launched in December 2023. Gemini models handle text, images, audio and video and power many Google products., alongside Jeff Dean and Oriol Vinyals.
Related
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- Attention Is All You NeedAttention Is All You Need, prop. n.: The 2017 Google research paper that introduced the transformer, the neural-network design behind ChatGPT, Claude, Gemini and nearly every other modern AI language system.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
- multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory.
- SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.
- GeminiGemini, prop. n.: Google's family of AI models and its chatbot assistant, launched in December 2023. Gemini models handle text, images, audio and video and power many Google products.
- Ashish VaswaniAshish Vaswani, prop. n.: An Indian-born computer scientist listed first on 'Attention Is All You Need', the 2017 paper that introduced the transformer, the design behind today's language models.
Sources
- Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.
- Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv preprint.
Mentioned in
Attention Is All You NeedAttention Is All You Need, prop. n.: The 2017 Google research paper that introduced the transformer, the neural-network design behind ChatGPT, Claude, Gemini and nearly every other modern AI language system., Character.AICharacter.AI, prop. n.: A popular app where people chat with AI characters — invented personas, fictional figures or imitations of real people — that other users have created., Google BrainGoogle Brain, prop. n.: Google's deep-learning research team from 2011 to 2023, where the transformer — the design behind today's chatbots — and the TensorFlow software were created., Ashish VaswaniAshish Vaswani, prop. n.: An Indian-born computer scientist listed first on 'Attention Is All You Need', the 2017 paper that introduced the transformer, the design behind today's language models., Jakob UszkoreitJakob Uszkoreit, prop. n.: A German computer scientist, one of the eight authors of the transformer paper, credited with first proposing to build language models on attention alone.