deep learning
In plain wordsMachine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract.
The branch of machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats. that trains artificial neural networksneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. with many layerslayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage. and lets them discover for themselves the intermediate representations a task requires, instead of relying on featuresfeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself. designed by people. Because the layers are stacked, each can build on the representations of the one below, from simple local patterns near the input (a contrast boundary in an image, a burst of sound) to abstract structure deeper in (a face, a spoken word, the sense of a sentence). The parameters, from millions to hundreds of billions of them, are learned jointly by backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. and gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. on large datasets, typically on GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. or other accelerators. Characteristic architectures include convolutionalconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects., recurrentrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. and transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. networks. From the early 2010s deep learning displaced hand-engineered features in vision, speech and language, and it underlies modern generative AIgenerative AI, n.: AI that produces something new on request, such as an essay, a picture, a song or a piece of code, having absorbed from enormous numbers of existing examples what such things typically look like.. Its costs are a large appetite for data and compute and limited transparency (the black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer. problem); see also representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand..
Deep learning only beat the hand-built speech pipeline once we had a thousand hours of transcribed audio.
Origin
Rina Dechter introduced the phrase to machine learning in 1986 (for a constraint-satisfaction search technique); Igor Aizenberg and colleagues applied it to neural networks in 2000. The modern sense spread after Geoffrey Hinton and colleagues' 2006 work on deep belief networks and became mainstream after AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward. won the 2012 ImageNet challenge. Hinton, Yoshua Bengio and Yann LeCun shared the 2018 Turing Award for the field.
Usage note
Priority is disputed: Jürgen Schmidhuber and others trace deep, multilayer learning back to Alexey Ivakhnenko's work in the 1960s. 'Deep' refers to the number of layers, not depth of understanding; a network with a single hidden layer is usually called shallow.
Related
- neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain.
- machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
- deep learning revolutiondeep learning revolution, n.: The rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward.
- godfathers of AIgodfathers of AI, phr.: A media nickname for three scientists — Geoffrey Hinton, Yoshua Bengio and Yann LeCun — whose decades of work on neural networks underpins modern AI. They shared computing's top prize in 2019.
- symbolic AIsymbolic AI, n.: An approach to AI that writes knowledge down as human-readable symbols and rules, such as 'every bird has wings', and reasons by manipulating them step by step instead of learning patterns from data.
- feature engineeringfeature engineering, n.: Turning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'.
Sources
- LeCun, Y., Bengio, Y. & Hinton, G. (2015). Deep learning. Nature, 521, 436–444.
- Hinton, G. E., Osindero, S. & Teh, Y.-W. (2006). A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18(7), 1527–1554.
- Goodfellow, I., Bengio, Y. & Courville, A. (2016). Deep Learning. MIT Press.
Mentioned in
artificial intelligenceartificial intelligence, n.: Computer systems that do things we would normally credit to a thinking mind, such as holding a conversation, reading an X-ray or planning a route, and the field of research that builds them., deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations., deepfakedeepfake, n.: A fake but realistic video, photo or voice recording made with AI that puts words in a real person's mouth or shows them in a scene that never happened., facial recognitionfacial recognition, n.: Technology that identifies or checks who someone is by comparing a picture of their face with stored photos, used by phones, passport gates and police databases., Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers., GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., Jensen HuangJensen Huang, prop. n.: The Taiwanese-American co-founder and chief executive of Nvidia, whose graphics chips became the main engine for training and running modern AI., layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage., machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats., neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., NvidiaNvidia, prop. n.: An American chip company whose graphics processors became the main hardware for training and running AI, making it one of the most valuable companies in the world., roboticsrobotics, n.: The science and engineering of building robots and making them work: designing their bodies, sensors and motors, and writing the software that lets them see, plan and move safely., symbolic AIsymbolic AI, n.: An approach to AI that writes knowledge down as human-readable symbols and rules, such as 'every bird has wings', and reasons by manipulating them step by step instead of learning patterns from data., AlphaFoldAlphaFold, prop. n.: A DeepMind AI system that predicts the 3D shape of proteins from their chemical sequence, often about as accurately as laboratory experiments. It cracked a problem biologists had worked on for 50 years and won a Nobel Prize., Andrew NgAndrew Ng, prop. n.: A British-born American computer scientist whose online courses have taught machine learning to millions; he also co-founded Google's deep learning team, Google Brain, and the online-course company Coursera., BaiduBaidu, prop. n.: China's largest search-engine company and an early investor in deep learning, which makes the Ernie chatbot and models and runs driverless taxis., big databig data, n.: Data that piles up too fast, in too many forms or in too great a quantity for one ordinary computer to store and study, and the methods invented to cope with it., black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer., connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols., CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math., deep learning revolutiondeep learning revolution, n.: The rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems., deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images., DeepMindDeepMind, prop. n.: A London artificial intelligence company, bought by Google in 2014, famous for programs that mastered the game of Go and predicted the shapes of proteins. In 2023 it merged into Google DeepMind.