pruning
In plain wordsCutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well.
- 1
Neural networks
The deletion of individual parametersparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned. or whole components from a trained neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., accepting a small loss of accuracy in exchange for a smaller or cheaper model; a short round of retraining usually wins most of the accuracy back. Unstructured pruning zeroes individual weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself., typically those with the smallest magnitudes or the least estimated effect on the loss, leaving a sparse network that needs sparsesparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation. storage and kernels to run faster; semi-structured N:M sparsityN:M sparsity, n.: A tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster. keeps a hardware-friendly pattern; structured pruning removes entire neurons, channels, attention headsattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., experts or layers, producing a smaller dense model that runs faster on ordinary hardware. Saliency criteria range from weight magnitude to second-order estimates based on the HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.. Large language models can be pruned in one shot: SparseGPT removed half or more of the weights of 175-billion-parameter models in a few hours without retraining. Pruning is often combined with quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. and knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run..
After pruning 40 percent of the channels and fine-tuning for a day, the detector ran twice as fast on the phone.
- 2
Decision trees
Simplifying a decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer. by removing branches that add little predictive value, to reduce overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.. Post-pruning grows a full tree and then collapses subtrees, as in cost-complexity pruning, which trades training error against tree size with a penalty chosen by cross-validation, or reduced-error pruning on held-out data; pre-pruning stops growth early with limits on depth or minimum gain.
- 3
Search
Skipping parts of a search tree that cannot change the result, as in alpha–beta pruningalpha–beta pruning, n.: A shortcut for game programs: tracking the best result each side is already sure of, it ignores lines of play that could only do worse, and when promising moves come first it searches about twice as deep. for game playing: the same metaphor of cutting branches, applied to a search procedure rather than to a learned model.
Origin
The gardening metaphor was established in game-tree search by the 1970s (alpha–beta pruning). For decision trees, CART (Leo Breiman, Jerome Friedman, Richard Olshen and Charles Stone, 1984) established cost-complexity pruning. For neural networks, Jocelyn Sietsma and Robert Dow's 'Neural net pruning: why and how' (1988) was soon followed by 'Optimal Brain Damage' (Yann LeCun, John Denker and Sara Solla, 1989) and 'Optimal Brain Surgeon' (Babak Hassibi and David Stork, 1993), which used second-order saliency; Song Han and colleagues revived magnitude pruning for deep networks in 2015.
Usage note
Pruning a trained network differs from training a small one from scratch, although the lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket. suggests that well-trainable sparse subnetworks exist from the start. Not to be confused with data pruning, which removes training examples rather than parameters.
Related
- sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
- N:M sparsityN:M sparsity, n.: A tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster.
- model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
- lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket.
- alpha–beta pruningalpha–beta pruning, n.: A shortcut for game programs: tracking the best result each side is already sure of, it ignores lines of play that could only do worse, and when promising moves come first it searches about twice as deep.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
Sources
- LeCun, Y., Denker, J. S. & Solla, S. A. (1990). Optimal Brain Damage. Advances in Neural Information Processing Systems 2 (NIPS 1989).
- Breiman, L., Friedman, J. H., Olshen, R. A. & Stone, C. J. (1984). Classification and Regression Trees. Wadsworth.
- Frantar, E. & Alistarh, D. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. arXiv:2301.00774. arxiv.org/abs/2301.00774
Mentioned in
knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer., model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation., C4.5C4.5, prop. n.: A widely used 1990s program that learns decision trees and rule sets from data; unlike its predecessor, it trims away unhelpful branches, copes with gaps in the data and can split on measured numbers., CARTCART, abbr.: A classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise., Core MLCore ML, prop. n.: Apple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud., early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs., Fisher informationFisher information, n.: A number, or table of numbers, saying how sharply the probability of the data reacts when a model's settings are nudged; the sharper the reaction, the more precisely the data can pin those settings down., lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket., N:M sparsityN:M sparsity, n.: A tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster., TinyMLTinyML, n.: Machine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years.