LeNet
In plain wordsA family of early image-recognition neural networks built at Bell Labs from 1989 that learned to read handwritten digits. A version read huge numbers of bank checks, and its design underlies today's image models.
A line of convolutional neural networksconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. for reading handwritten characters, among the first CNNs trained end to end with backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction., developed between 1989 and 1998 by Yann LeCunYann LeCun, prop. n.: A French-American computer scientist who pioneered the neural networks that recognize images, led Meta's AI research for over a decade, and argues that chatbots alone will not reach human-level AI.'s group at AT&T Bell Laboratories. They combined convolutional layers with shared weights, subsampling (pooling) layers and fully connected layers, learning their feature detectors directly from pixels instead of relying on hand-designed features. The best-known version, LeNet-5 (1998), passed 32 × 32 images through seven layers with about 60,000 trainable parameters. Systems built on LeNet were deployed by NCR and others to read handwritten amounts on bank checks, and by the late 1990s were processing a substantial share of checks written in the United States (more than 10 percent, by LeCun's account). The architecture is the template that AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward. and later vision networks scaled up.
Implementing LeNet-5 on MNIST is a standard first exercise in deep-learning courses.
Origin
First described in LeCun and colleagues' 'Backpropagation Applied to Handwritten Zip Code Recognition' (Neural Computation, 1989) and summarized in LeCun, Léon Bottou, Yoshua BengioYoshua Bengio, prop. n.: A Canadian computer scientist who helped found deep learning, including ideas behind today's language models, and who has become a leading voice on the risks of advanced AI. and Patrick Haffner's 'Gradient-Based Learning Applied to Document Recognition' (Proceedings of the IEEE, 1998), which named LeNet-5.
Related
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- MNISTMNIST, prop. n.: A famous collection of 70,000 small images of handwritten digits used to test image-recognition programs. Getting a model to read MNIST digits is the 'hello world' of machine learning.
- neocognitronneocognitron, n.: A 1980 neural network for recognizing handwritten characters, modeled on the brain's visual system. Its stacked layers of pattern detectors were the blueprint for the networks used in modern image recognition.
- AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward.
- Yann LeCunYann LeCun, prop. n.: A French-American computer scientist who pioneered the neural networks that recognize images, led Meta's AI research for over a decade, and argues that chatbots alone will not reach human-level AI.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
Sources
- LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W. & Jackel, L. D. (1989). Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation 1(4), 541–551.
- LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE 86(11), 2278–2324.
Mentioned in
Yann LeCunYann LeCun, prop. n.: A French-American computer scientist who pioneered the neural networks that recognize images, led Meta's AI research for over a decade, and argues that chatbots alone will not reach human-level AI., AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward., convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects., MNISTMNIST, prop. n.: A famous collection of 70,000 small images of handwritten digits used to test image-recognition programs. Getting a model to read MNIST digits is the 'hello world' of machine learning., Bell LabsBell Labs, prop. n.: A famous American industrial research laboratory, long owned by the telephone company AT&T, where the transistor, information theory and key early machine-learning methods were invented., neocognitronneocognitron, n.: A 1980 neural network for recognizing handwritten characters, modeled on the brain's visual system. Its stacked layers of pattern detectors were the blueprint for the networks used in modern image recognition.