deep learning revolution
In plain wordsThe rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems.
The period beginning around 2010–2012 in which deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract. — training neural networks with many layers on large datasets using GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. — rapidly displaced other methods across artificial intelligence. Milestones include sharp cuts in speech-recognition error by deep networks developed by Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers.'s group with Microsoft, Google and IBM (2009–2012); AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward.'s victory in the 2012 ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom. challenge; word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text. (2013), sequence-to-sequence learning (2014) and Google's switch to neural machine translation (2016); AlphaGoAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it. (2016); and the transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. (2017), which led to large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.. The enabling conditions were cheap parallel computing, internet-scale data and algorithmic advances such as ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. activations, dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns. and better initialization and optimization. Technology companies hired academic researchers en masse and bought start-ups such as DeepMind (2014).
Most of today's AI laboratories are institutions built during the deep learning revolution.
Origin
The phrase was common in technology journalism by the mid-2010s and became the title of Terrence Sejnowski's book The Deep Learning Revolution (MIT Press, 2018). It echoes Thomas Kuhn's idea of a scientific revolution as a paradigm shift.
Usage note
The start date is conventional: the methods developed over decades, with earlier milestones such as the deep belief networkdeep belief network, n.: A many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it. (2006) and GPU-trained handwriting networks (2010–2011). The 2018 Turing AwardTuring Award, prop. n.: Nicknamed the 'Nobel Prize of computing', the top honor in computer science, awarded each year since 1966 by the Association for Computing Machinery. Several AI pioneers have received it. and the 2024 Nobel Prizes for AINobel Prizes for AI, phr.: The two 2024 Nobel Prizes, in physics and chemistry, that honored artificial intelligence research for the first time, recognizing neural-network pioneers and the protein-structure system AlphaFold. recognized its pioneers.
Related
- deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract.
- AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward.
- ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom.
- godfathers of AIgodfathers of AI, phr.: A media nickname for three scientists — Geoffrey Hinton, Yoshua Bengio and Yann LeCun — whose decades of work on neural networks underpins modern AI. They shared computing's top prize in 2019.
- the bitter lessonthe bitter lesson, phr.: The argument, made by AI pioneer Rich Sutton in 2019, that AI has advanced mostly through general methods that use ever more computing power, not by programming in human knowledge.
- connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols.
Sources
- Sejnowski, T. J. (2018). The Deep Learning Revolution. Cambridge, MA: MIT Press.
- LeCun, Y., Bengio, Y. & Hinton, G. (2015). Deep Learning. Nature 521, 436–444.
Mentioned in
deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract., Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers., GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward., godfathers of AIgodfathers of AI, phr.: A media nickname for three scientists — Geoffrey Hinton, Yoshua Bengio and Yann LeCun — whose decades of work on neural networks underpins modern AI. They shared computing's top prize in 2019., ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom., the bitter lessonthe bitter lesson, phr.: The argument, made by AI pioneer Rich Sutton in 2019, that AI has advanced mostly through general methods that use ever more computing power, not by programming in human knowledge., Turing AwardTuring Award, prop. n.: Nicknamed the 'Nobel Prize of computing', the top honor in computer science, awarded each year since 1966 by the Association for Computing Machinery. Several AI pioneers have received it., Alex KrizhevskyAlex Krizhevsky, prop. n.: A Canadian computer scientist who built AlexNet, the image-recognition network that was trained on gaming graphics chips and whose 2012 contest win launched the deep learning boom., deep belief networkdeep belief network, n.: A many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it., GPGPUGPGPU, abbr.: Using a graphics chip for ordinary number-crunching instead of drawing pictures — the trick that, years later, made modern AI training practical., hardware lotteryhardware lottery, n.: The idea that research ideas succeed partly by luck — because they happen to suit the computer chips available at the time — rather than purely because they are better., Terry SejnowskiTerry Sejnowski, prop. n.: An American neuroscientist who co-invented the Boltzmann machine with Geoffrey Hinton, built an early network that learned to read aloud, and helped found the NeurIPS conference community.