inductive bias
In plain wordsThe assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.
The prior commitments of a learning algorithm — about which functions are possible and which are more plausible — that let it choose between hypotheses its training data cannot tell apart, and so shape how it extends to new inputs. Inductive bias can restrict what can be represented (a linear model cannot express XOR) or rank hypotheses within the space (shorter trees, smaller weights, larger marginsmargin, n.: How far, and on which side, an example lies from a classifier's dividing line. A wide gap between the line and the nearest examples suggests the classifier will handle new data well.). Examples include smoothness and locality in k-nearest neighborsk-nearest neighbors, n.: Predicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average., conditional independence in naive Bayesnaive Bayes, n.: A simple, fast way of sorting items into categories using probabilities, which pretends that each clue is independent of the others. It is the classic technique behind early email spam filters., axis-aligned splits in decision treesdecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer., locality and translation equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way. in convolutional networksconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects., and the comparatively weak biases of transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., which compensate with data. The no free lunch theoremno free lunch theorem, n.: A mathematical result saying that no learning or search method is best for every possible problem: any method that wins on some problems must lose on others. shows why some bias is unavoidable: averaged over all possible problems, no learner beats any other. Choosing an architecture, a regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. scheme or a priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns. is choosing an inductive bias.
Convolutions bake in a useful inductive bias for images, so the CNN needed far less data than the vision transformer.
Origin
The notion was introduced, as 'bias', by Tom Mitchell in the Rutgers technical report 'The Need for Biases in Learning Generalizations' (1980), which argued that an unbiased learner cannot generalize beyond its examples; Paul Utgoff's 1984 thesis studied shifting bias, and his 1986 book was titled 'Machine Learning of Inductive Bias'. The underlying problem echoes David Hume's problem of induction and Nelson Goodman's 'new riddle of induction' (1955).
Usage note
Unrelated to bias in the fairness sense (algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.) and distinct from a neuron's bias term or an estimator's statistical bias; see the hub entry biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system..
Related
- no free lunch theoremno free lunch theorem, n.: A mathematical result saying that no learning or search method is best for every possible problem: any method that wins on some problems must lose on others.
- generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.
- Occam's razorOccam's razor, n.: A rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise.
- bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
- equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way.
- the bitter lessonthe bitter lesson, phr.: The argument, made by AI pioneer Rich Sutton in 2019, that AI has advanced mostly through general methods that use ever more computing power, not by programming in human knowledge.
- algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.
Sources
- Mitchell, T. M. (1980). The Need for Biases in Learning Generalizations. Technical report, Rutgers University, New Brunswick, NJ.
Mentioned in
architecturearchitecture, n.: The overall design of an AI model, meaning what kinds of parts it has and how they are connected, as opposed to the specific numbers it learns during training., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much., fully connected layerfully connected layer, n.: A layer in which every input is connected to every output by its own adjustable weight. It is the most general and most common kind of neural-network layer., inductive reasoninginductive reasoning, n.: Reasoning from particular examples to a general rule or prediction — every swan I have seen is white, so swans are probably white — which is useful but can always turn out wrong., multi-task learningmulti-task learning, n.: Training one model to do several related jobs at once, such as spotting objects and judging distance in the same photo, so that practice on each job can sharpen the others., parametric modelparametric model, n.: A model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones., priorprior, n.: What you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns., tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this., universal approximation theoremuniversal approximation theorem, n.: A mathematical result showing that a neural network with enough neurons can imitate essentially any smooth input-to-output relationship as closely as you like, though it does not say how to find that network., weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data., disentangled representationdisentangled representation, n.: An internal representation in which each part tracks one separate, meaningful factor of the data, such as one number for a face's angle, another for its lighting and another for hair color., equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way., geometric deep learninggeometric deep learning, n.: An approach to designing neural networks around the shapes and symmetries of the data, whether grids, graphs, surfaces or 3-D structures, so the network respects how that data can be moved or rearranged., hypothesis spacehypothesis space, n.: The full menu of possible models a learning method is allowed to choose from — for example, all straight lines. Learning means picking the item on the menu that best fits the data., ID3ID3, abbr.: An early, influential method for automatically building a decision tree from examples, choosing at each step the question that most reduces uncertainty about the answer., implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions., invarianceinvariance, n.: A property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped., no free lunch theoremno free lunch theorem, n.: A mathematical result saying that no learning or search method is best for every possible problem: any method that wins on some problems must lose on others., Tom MitchellTom Mitchell, prop. n.: An American computer scientist who wrote a classic machine-learning textbook with the field's most quoted definition of learning, and founded the first university department devoted to machine learning.