double descent
In plain wordsA surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.
The phenomenon in which test error, plotted against model capacity, first follows the classical U-shaped curve of the bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between. — falling, then rising as the model starts fitting noise — peaks near the interpolation threshold, where capacity is just enough to fit the training data exactly, and then falls again as capacity grows further, often below the first minimum. Near the threshold essentially one fit exists and it is badly conditioned; well beyond it, many interpolating solutions exist and training with implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions. picks smooth, low-norm ones. The pattern appears in linear regression, random-feature models, tree ensembles and deep networks, and also as a function of training time (epoch-wise double descent) and of dataset size, where adding data can temporarily hurt. The peak is most pronounced with label noise and weak regularization, and well-tuned regularization can remove it. Double descent helped explain why very large, overparameterizedoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps. networks generalize well.
Origin
Peaks in test error at the interpolation threshold were observed in statistical-physics studies of learning from 1989 onward, but the name was coined by Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal (arXiv December 2018; PNAS 2019). Preetum Nakkiran and colleagues' 'Deep Double Descent' (2019) extended it to deep networks, training time and dataset size.
Related
- overparameterizationoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps.
- bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
- benign overfittingbenign overfitting, n.: When a model fits its training data perfectly — even the errors and noise — yet still predicts well on new data, defying the textbook warning that this always backfires.
- implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- generalization gapgeneralization gap, n.: The difference between how well a model does on the examples it trained on and how well it does on new ones. A big gap means it memorized rather than learned.
Sources
- Belkin, M., Hsu, D., Ma, S. & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS 116(32), 15849–15854. arxiv.org/abs/1812.11118
- Nakkiran, P. et al. (2020). Deep Double Descent: Where Bigger Models and More Data Hurt. ICLR 2020. arxiv.org/abs/1912.02292
- Loog, M., Viering, T., Mey, A., Krijthe, J. H. & Tax, D. M. J. (2020). A brief prehistory of double descent. PNAS 117(20), 10625–10626.
Mentioned in
overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between., early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples., generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers., learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help., model capacitymodel capacity, n.: How much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise., Occam's razorOccam's razor, n.: A rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise., benign overfittingbenign overfitting, n.: When a model fits its training data perfectly — even the errors and noise — yet still predicts well on new data, defying the textbook warning that this always backfires., grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right., implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions., overparameterizationoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps., statistical learning theorystatistical learning theory, n.: The mathematics of when and why learning from examples works: how much data is needed, and how well a model trained on some examples will do on new ones., VC dimensionVC dimension, n.: A measure of how flexible a family of prediction rules is: the largest number of points it can label in every possible way. More flexible families need more data to learn reliably.