Skip to content
Last Dictionary
Plane
Machine LearningMathematics & StatisticsDeep Learning

bias–variance tradeoff

nounAlso: bias–variance trade-off, bias–variance decomposition, bias–variance dilemma

In plain wordsThe balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.

  1. The decomposition of a model's expected prediction error on new data into squared bias (how far the average fitted model, over many possible training sets, sits from the truth), variance (how much the fitted model changes from one training set to another) and irreducible noise, together with the observation that, for a fixed amount of data, increasing model flexibility typically lowers bias but raises variance. Classically, test error plotted against model complexity is U-shaped, with underfittingunderfitting, n.: When a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from. on one side and overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding. on the other; regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error. (mainly reducing variance) and boostingboosting, n.: Building a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made. (mainly reducing bias) move along or around this curve. The squared-error decomposition is exact; for 0–1 classification loss several competing decompositions exist. Heavily overparameterized neural networks complicate the picture: test error can fall again as capacity grows past the point of fitting the training data exactly (double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.), so the U-curve is now seen as one regime rather than a universal law.

    Adding polynomial terms cut bias but the variance blew up, so we settled on degree three.

E[(y−f^(x))2]=Bias⁡[f^(x)]2+Var⁡[f^(x)]+σ2\mathbb{E}\big[(y - \hat{f}(\mathbf{x}))^2\big] = \operatorname{Bias}\big[\hat{f}(\mathbf{x})\big]^2 + \operatorname{Var}\big[\hat{f}(\mathbf{x})\big] + \sigma^2

Origin

The split of mean squared error into bias and variance is long-standing statistics; Stuart Geman, Elie Bienenstock and René Doursat's 'Neural Networks and the Bias/Variance Dilemma' (Neural Computation, 1992) made it central to machine learning. Ron Kohavi and David Wolpert (1996) and Pedro Domingos (2000) extended it to classification, and Mikhail Belkin and colleagues described double descent in 2019.

Usage note

'Bias' here is the statistical bias of an estimator, not algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.. Brady Neal and others have argued that textbooks overstate the tradeoff for neural networks, where wider models often show lower variance.

Related

See also
  • overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
  • underfittingunderfitting, n.: When a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from.
  • regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
  • double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.
  • generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.
  • model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
  • variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves.

Sources

  1. Geman, S., Bienenstock, E. & Doursat, R. (1992). Neural Networks and the Bias/Variance Dilemma. Neural Computation, 4(1), 1–58.
  2. Belkin, M., Hsu, D., Ma, S. & Mandal, S. (2019). Reconciling Modern Machine-Learning Practice and the Classical Bias–Variance Trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849–15854.

Mentioned in

algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records., overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases., learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help., mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better., model capacitymodel capacity, n.: How much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise., model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best., parametric modelparametric model, n.: A model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones., polynomial regressionpolynomial regression, n.: Fitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., underfittingunderfitting, n.: When a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from., variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves., Bayes error rateBayes error rate, n.: The lowest error any classifier could ever achieve on a task, because some cases are genuinely ambiguous: even a perfect model that knew the true odds would still sometimes guess wrong., double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse., irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence., statistical learning theorystatistical learning theory, n.: The mathematics of when and why learning from examples works: how much data is needed, and how well a model trained on some examples will do on new ones.