Skip to content
Last Dictionary
Plane
Data & DatasetsMachine LearningMathematics & Statistics

covariate shift

nounAlso: input distribution shift

In plain wordsA kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.

  1. A form of distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested. in which the distribution of inputs (covariates) differs between training and test data, p_train(x) ≠ p_test(x), while the conditional distribution of outputs given inputs, p(y | x), is unchanged. A model that had learned p(y | x) correctly everywhere would be unaffected, but real models are fitted mostly where training data are dense, so errors grow in regions that become common at test time. The classical correction is importance weighting: training examples are reweighted by the ratio of test to training input densities, estimated directly with methods such as kernel mean matching or KLIEP, or with a classifier trained to tell training inputs from test inputs. Weighting makes estimates high-variance when the two distributions overlap little, and no reweighting helps where test inputs fall outside the training data's support.

    The new scanner produced brighter images, a covariate shift our importance weights only partly corrected.

w(x)=ptest(x)ptrain(x)w(x) = \frac{p_{\text{test}}(x)}{p_{\text{train}}(x)}

Origin

The term is usually traced to Hidetoshi Shimodaira's 'Improving Predictive Inference under Covariate Shift by Weighting the Log-Likelihood Function' (Journal of Statistical Planning and Inference, 2000), which analyzed importance weighting under such shifts; Masashi Sugiyama and colleagues developed density-ratio estimation methods for it in the late 2000s.

Usage note

Contrast label shiftlabel shift, n.: A kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before., where class proportions change but each class's inputs look the same, and concept shift, where p(y | x) itself changes. In MLOps, input changes over time are monitored as data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable..

Related

See also
  • distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
  • data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable.
  • out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
  • sampling biassampling bias, n.: A distortion that arises when the data collected over-represents some kinds of people or situations and under-represents others, so conclusions drawn from it don't hold for everyone.
  • transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
Contrast with
  • label shiftlabel shift, n.: A kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before.
  • concept driftconcept drift, n.: When the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold.

Sources

  1. Shimodaira, H. (2000). Improving Predictive Inference under Covariate Shift by Weighting the Log-Likelihood Function. Journal of Statistical Planning and Inference 90(2), 227–244.

Mentioned in

concept driftconcept drift, n.: When the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold., data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable., distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested., domain adaptationdomain adaptation, n.: Adjusting a model trained on one kind of data, such as daytime photos or one hospital's records, so that it still works on a different but related kind, such as night photos or another hospital's., out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable., DAggerDAgger, n.: A way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes., label shiftlabel shift, n.: A kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before.