data drift
In plain wordsWhen the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable.
A change over time in the statistical distribution of a deployed model's input data relative to its training or reference data; the production-monitoring counterpart of covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.. Causes include changes in the user population, seasonality, new products or markets, upstream pipeline or sensor changes, and adversaries adapting to the model. Drift is detected by comparing recent windows with a reference: per-feature tests and distances (population stability index, Kolmogorov–Smirnov, chi-squared, Jensen–Shannon or Wasserstein distance), multivariate detectors, classifiers trained to tell old data from new, or, for text and images, distances between embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.. Because a change in inputs need not change the input–output relationship, drift may or may not reduce accuracy; it is a warning that prompts evaluation on fresh labels, retraining or recalibration. Upstream bugs, such as a field silently switching units, often surface first as apparent drift.
The data drift alert fired after the app update started sending transaction amounts in cents.
Origin
The label spread with MLOps tooling in the late 2010s, by analogy with concept driftconcept drift, n.: When the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold. (1986). The phenomenon had long been studied under other names, including population drift in credit scoring, where the population stability index is a long-standing check, and covariate shift in statistics (Hidetoshi Shimodaira, 2000).
Usage note
Usage is inconsistent: some authors call any distribution change data drift, including concept drift. Strictly, data drift concerns P(x) and concept drift concerns P(y | x). distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested. is the umbrella term in research.
Related
- concept driftconcept drift, n.: When the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold.
- covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.
- training–serving skewtraining–serving skew, n.: A mismatch between how data is prepared when a model is trained and when it is used for real, which makes the live model quietly worse than its tests promised.
- out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
- concept driftconcept drift, n.: When the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold.
Mentioned in
concept driftconcept drift, n.: When the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold., distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested., MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself., model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm., model risk managementmodel risk management, n.: The rules banks and other firms follow to make sure the mathematical models and AI systems they rely on are built correctly, checked independently and used only for what they are fit for., covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same., machine learning technical debtmachine learning technical debt, n.: The hidden future cost of quick shortcuts in machine-learning systems — tangled data dependencies, messy glue code, unchecked feedback loops — that later make them hard and risky to change., post-market monitoringpost-market monitoring, n.: The duty of an AI system's maker to keep watching how it performs after release, gathering evidence from real use so that problems are found and fixed., training–serving skewtraining–serving skew, n.: A mismatch between how data is prepared when a model is trained and when it is used for real, which makes the live model quietly worse than its tests promised.