imputation
In plain wordsFilling in missing values in a dataset with sensible estimates, such as the typical value or a prediction from the other columns, so the data can be analyzed or used for training.
The filling-in of missing entries in a dataset with estimated values so that analysis or training can use incomplete records. Simple methods fill each gap with a constant, the mean, median or mode, or, in time series, the last observed value; model-based methods predict missing entries from other variables using regression, k-nearest neighborsk-nearest neighbors, n.: Predicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average., random forestsrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree., matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together., the expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes. or deep generative models. Multiple imputation creates several completed datasets by drawing plausible values, analyzes each, and pools the results so that uncertainty about the missing values carries into the final estimates; chained-equation implementations (MICE) model each incomplete variable conditionally on the others. Validity depends on why values are missing: when data are missing completely at random, or at random given the observed variables, imputation models that condition on the observed data can avoid bias (simple mean filling still distorts variances and correlations), but data missing not at random, where missingness depends on the unobserved value itself, cannot be repaired by imputation alone. Adding a binary 'was missing' indicator is common practice.
We imputed missing blood-pressure readings with the median and added an indicator column, which itself proved predictive.
Origin
Long used informally in survey statistics. Donald Rubin's 1976 Biometrika paper 'Inference and Missing Data' provided the theoretical framework, including the notion of data missing at random, and his multiple imputation, proposed in the late 1970s, was set out in his 1987 book Multiple Imputation for Nonresponse in Surveys. Stef van Buuren's MICE software popularized chained-equation imputation.
Usage note
Imputed values are estimates, not observations: single imputation understates uncertainty, and imputation statistics must be learned from training data only, or test information leaks in (data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.).
Related
- data cleaningdata cleaning, n.: Finding and fixing errors in a dataset — typos, duplicates, missing values, impossible numbers, wrong labels — before using it, because flawed data produces flawed results.
- expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes.
- feature engineeringfeature engineering, n.: Turning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'.
- data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.
- k-nearest neighborsk-nearest neighbors, n.: Predicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average.
- data qualitydata quality, n.: How well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills.
Sources
- Rubin, D. B. (1976). Inference and Missing Data. Biometrika 63(3).
- Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. New York: Wiley.
- van Buuren, S. & Groothuis-Oudshoorn, K. (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software 45(3).
Mentioned in
data cleaningdata cleaning, n.: Finding and fixing errors in a dataset — typos, duplicates, missing values, impossible numbers, wrong labels — before using it, because flawed data produces flawed results., expectation–maximization algorithmexpectation–maximization algorithm, n.: A method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes., feature engineeringfeature engineering, n.: Turning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'.