fairness impossibility theorem
In plain wordsA mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.
Any of several closely related results showing that common group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean. criteria are mutually incompatible except in degenerate cases. Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan proved that a risk score cannot simultaneously be calibrated within each group and balanced for both the positive and the negative class (equal average scores for true positives, and for true negatives, across groups) unless base rates are equal or prediction is perfect. Alexandra Chouldechova showed that if base rates differ, a classifier with equal positive predictive value across groups (predictive paritypredictive parity, n.: A fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group.) must have unequal false positive or false negative rates, violating equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often.. The results explain how both sides of the COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous. dispute could be right about different metrics, and they force an explicit choice about which errors to equalize.
Given the impossibility theorem, the committee had to decide whether equal error rates or equally meaningful scores mattered more.
Origin
Proved independently in 2016, prompted by the COMPAS controversy: Kleinberg, Mullainathan and Raghavan, 'Inherent Trade-Offs in the Fair Determination of Risk Scores' (arXiv September 2016; ITCS 2017), and Chouldechova, 'Fair prediction with disparate impact' (arXiv October 2016; Big Data, 2017). Sorelle Friedler, Carlos Scheidegger and Suresh Venkatasubramanian argued along related lines the same year.
Usage note
The name is informal: there is no single theorem but a family of results. They do not show that fairness is impossible, only that certain statistical criteria cannot all be equalized when base rates differ.
Related
- predictive paritypredictive parity, n.: A fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group.
- equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often.
- demographic paritydemographic parity, n.: A fairness test that asks whether a system says yes (approves the loan, shortlists the applicant) at the same rate for every group, whatever other differences exist between them.
- COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous.
- calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- algorithmic fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not.
Sources
- Kleinberg, J., Mullainathan, S. & Raghavan, M. (2017). Inherent Trade-Offs in the Fair Determination of Risk Scores. Proc. ITCS 2017. arxiv.org/abs/1609.05807
- Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data 5(2), 153–163. arxiv.org/abs/1610.07524
Mentioned in
algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records., algorithmic fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous., demographic paritydemographic parity, n.: A fairness test that asks whether a system says yes (approves the loan, shortlists the applicant) at the same rate for every group, whatever other differences exist between them., equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often., group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean., predictive paritypredictive parity, n.: A fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group.