cross-validation
In plain wordsA fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.
A model-evaluation technique that reuses one dataset: on each round a model is trained on part of the data and scored on the withheld remainder, and the pooled scores estimate how well the modeling procedure generalizes or rank competing models and hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.. In k-fold form the data are cut into k folds (usually 5 or 10), each withheld exactly once; the k scores are averaged, their spread indicating variability. Leave-one-out uses as many folds as examples. Variants respect the structure of the data: stratified folds preserve class proportions, grouped folds keep all records of one patient or user together, and time-series cross-validation always trains on the past and validates on the future, using rolling or expanding windows. Nested cross-validation runs an inner tuning loop inside an outer evaluation loop so that the reported estimate is not biased by tuning. Every preprocessing step must be fitted within the training folds to avoid data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.. Because it costs k training runs, cross-validation is routine for classical models but rare for large neural networks, which use a single validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one..
Five-fold cross-validation put the model at 0.81 ± 0.02 AUC, so the 0.83 holdout score was no fluke.
Origin
Early forms appear in psychological testing from the 1930s; Charles Mosier's 1951 paper discussed designs for cross-validation. Mervyn Stone (1974) and Seymour Geisser (1975) set out the modern theory of cross-validation for prediction and model choice, and Ron Kohavi's 1995 study popularized stratified 10-fold cross-validation in machine learning.
Usage note
Cross-validation evaluates a modeling procedure rather than one fitted model: after choosing settings, refit on all training data. Tuning and reporting with the same folds gives optimistic scores.
Related
- validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one.
- training settraining set, n.: The examples a machine-learning model learns from, like the worked problems a student practices on, kept separate from the examples later used to check how well it learned.
- test settest set, n.: A set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam.
- model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.
- bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
Sources
- Stone, M. (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society, Series B 36(2).
- Geisser, S. (1975). The Predictive Sample Reuse Method with Applications. Journal of the American Statistical Association 70(350).
- Kohavi, R. (1995). A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. Proc. IJCAI 1995.
Mentioned in
overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., test settest set, n.: A set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam., training settraining set, n.: The examples a machine-learning model learns from, like the worked problems a student practices on, kept separate from the examples later used to check how well it learned., bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data., data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints., feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate., grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science., model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way., time series forecastingtime series forecasting, n.: Predicting future values of something measured over time, such as sales, electricity demand, prices or rainfall, from its past values and related information., validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one., Akaike information criterionAkaike information criterion, n.: A score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better., CARTCART, abbr.: A classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise., elastic netelastic net, n.: A blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random., stackingstacking, n.: Combining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen.