Occam's razor

A methodological preference, other things being equal, for explanations that posit fewer entities, assumptions or adjustable parts. In machine learning it appears as a preference for models of lower capacity or shorter description, justified by generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers. arguments: regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best. criteria such as AIC and BIC, minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress. and Bayesian model comparison (where the marginal likelihoodlikelihood, n.: A score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen. automatically penalizes needless flexibility) all implement versions of it, and PAC learningPAC learning, n.: A mathematical definition of what it means to learn from examples: given enough data, a learner should, with high probability, find a rule that is almost always right. theory includes an 'Occam's razor' theorem showing that learners which output short hypotheses generalize. Solomonoff inductionSolomonoff induction, n.: A mathematical recipe for ideal prediction: consider every computer program that could have produced the data so far, trust shorter programs more, and predict what they would output next. makes the principle precise by weighting hypotheses by program length. Deep learning complicates the picture: hugely overparameterizedoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps. networks generalize well, suggesting that simplicity should be measured by the functions training actually favors rather than by parameter count — an implicit simplicity bias studied under implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions. and double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse..

By Occam's razor, try the linear baseline first; the transformer has to earn its complexity.