concept drift
In plain wordsWhen the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold.
A change over time in the relationship between inputs and the target a model predicts, so that the conditional distribution P(y | x) at deployment differs from the one learned in training. A transaction pattern may stop indicating fraud once fraudsters adapt; words may carry different sentiment as language shifts; a pandemic may change what predicts demand. Drift may be sudden, gradual, incremental or recurring (as with seasons). Unlike data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable., it can occur while input distributions stay the same, so it is usually detected through falling accuracy on newly labeled data, error-rate monitoring, or stream-learning detectors such as DDM (Drift Detection Method) and ADWIN (adaptive windowing). Remedies include scheduled or triggered retraining, weighting recent data more heavily, sliding training windows, online learningonline learning, n.: Learning from data that arrive one piece at a time, updating the model after each new example instead of training once on a fixed collection, as a spam filter does while emails stream in., and ensembles that retire stale members.
After the policy change, concept drift made last year's approval model systematically too lenient.
Origin
Introduced by Jeffrey Schlimmer and Richard Granger in a 1986 paper on incremental learning from noisy data (the STAGGER system) in the journal Machine Learning. Gerhard Widmer and Miroslav Kubat's 1996 study of learning under concept drift and hidden contexts helped establish the topic, later central to data-stream mining and production MLOpsMLOps, n.: The working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself..
Usage note
In MLOps usage, any drift that reduces accuracy is often loosely called concept drift. The research literature separates it from covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same. and label shiftlabel shift, n.: A kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before..
Related
- data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.
- online learningonline learning, n.: Learning from data that arrive one piece at a time, updating the model after each new example instead of training once on a fixed collection, as a spam filter does while emails stream in.
- label shiftlabel shift, n.: A kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before.
- covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.
- data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable.
Sources
- Schlimmer, J. C. & Granger, R. H. (1986). Incremental Learning from Noisy Data. Machine Learning 1(3).
- Widmer, G. & Kubat, M. (1996). Learning in the Presence of Concept Drift and Hidden Contexts. Machine Learning 23.
Mentioned in
data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable., distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested., model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm., online learningonline learning, n.: Learning from data that arrive one piece at a time, updating the model after each new example instead of training once on a fixed collection, as a spam filter does while emails stream in., covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.