algorithmic bias
In plain wordsUnfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.
Systematic and unjustified differences in how an automated system treats, or performs for, different individuals or groups, especially along the lines of a protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly.. In the taxonomy of Harini Suresh and John Guttag, it can enter at each stage of building and using a model: historical bias (data that record past discrimination), representation bias (groups under-sampled in training data), measurement bias (a stand-in target such as arrests for crime, or health spending for medical need), aggregation and evaluation bias (one model or benchmark for heterogeneous populations) and deployment bias (use in settings the system was not built for). Bias can arise without intent and even when protected attributes are excluded, through proxy discriminationproxy discrimination, n.: Discrimination that happens when a system cannot see a trait like race directly but relies on something closely linked to it, such as a postcode, and so treats groups unequally anyway.. It is measured with fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not. metrics and reduced through better data, constrained training, post-processing and auditsAI audit, n.: An independent check of an AI system, or of the company using it, against agreed standards — for example, testing whether a hiring tool treats women and men fairly..
The audit found algorithmic bias: at equal risk scores, Black patients were sicker than white patients.
Origin
Batya Friedman and Helen Nissenbaum's 'Bias in Computer Systems' (1996) distinguished preexisting, technical and emergent bias. An early documented case was an admissions-screening program at St George's Hospital Medical School, London, found in 1988 to discriminate against women and applicants with non-European names. The term became widespread after ProPublica's 'Machine Bias' investigation of COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous. (2016).
Usage note
Distinct from the statistical sense of biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system. (systematic estimation error) and from the bias parameter of a neural network. Because common fairness metrics conflict (fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.), deciding which disparities count as bias is partly a value judgment.
Related
- algorithmic fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not.
- disparate impactdisparate impact, n.: When a rule or system that treats everyone alike on paper still lands much harder on one group, such as women or older workers; the law can forbid this even if no one intended it.
- proxy discriminationproxy discrimination, n.: Discrimination that happens when a system cannot see a trait like race directly but relies on something closely linked to it, such as a postcode, and so treats groups unequally anyway.
- COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous.
- Gender ShadesGender Shades, prop. n.: A 2018 study showing that commercial face-analysis software guessed the gender of darker-skinned women far less accurately than that of lighter-skinned men.
- protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly.
- fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.
- bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
Sources
- Friedman, B. & Nissenbaum, H. (1996). Bias in Computer Systems. ACM Transactions on Information Systems 14(3), 330–347.
- Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464), 447–453.
- Suresh, H. & Guttag, J. (2021). A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. Proc. EAAMO 2021.
Mentioned in
AI ethicsAI ethics, n.: The study of what is right and wrong in how AI is built and used: who it may harm, who is responsible when it fails, and which values it should respect., facial recognitionfacial recognition, n.: Technology that identifies or checks who someone is by comparing a picture of their face with stored photos, used by phones, passport gates and police databases., false positivefalse positive, n.: A false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there., AI auditAI audit, n.: An independent check of an AI system, or of the company using it, against agreed standards — for example, testing whether a hiring tool treats women and men fairly., AI literacyAI literacy, n.: Knowing enough about AI to use it sensibly and judge it critically: what it can and cannot do, how it goes wrong, and how it affects people., algorithmalgorithm, n.: A precise, step-by-step procedure for solving a problem or doing a task, like a recipe a computer can follow. In the news, 'the algorithm' often means software that ranks what you see online., algorithmic accountabilityalgorithmic accountability, n.: The idea that the people and organizations behind automated decision systems must be able to explain and justify those systems' effects, answer for the harm they cause, and fix it., algorithmic fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between., Blueprint for an AI Bill of RightsBlueprint for an AI Bill of Rights, prop. n.: A 2022 White House document listing five protections Americans should have when automated systems affect their lives — safety, fairness, privacy, explanation and a human alternative. It was guidance, not law., Colorado AI ActColorado AI Act, prop. n.: A Colorado law, first passed in 2024, governing AI used to make important decisions about people, such as hiring, lending and housing. In 2026 it was rewritten as a narrower law that starts in 2027., COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous., disparate impactdisparate impact, n.: When a rule or system that treats everyone alike on paper still lands much harder on one group, such as women or older workers; the law can forbid this even if no one intended it., face detectionface detection, n.: Finding where the faces are in a photo or video, without working out whose faces they are: the step that lets a camera focus on people or blur them out., Gender ShadesGender Shades, prop. n.: A 2018 study showing that commercial face-analysis software guessed the gender of darker-skinned women far less accurately than that of lighter-skinned men., inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases., Joy BuolamwiniJoy Buolamwini, prop. n.: A Canadian-American computer scientist and activist whose research showed that face-analysis software works far worse on darker-skinned women, and who founded the Algorithmic Justice League., predictive AIpredictive AI, n.: AI used to forecast outcomes about people or events, such as who will repay a loan, which patient may get sicker or which machine will fail, in order to guide decisions., predictive policingpredictive policing, n.: Using computer programs to forecast where crimes are likely to happen, or who is likely to be involved, so that police can decide where to patrol or whom to watch., protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly., proxy discriminationproxy discrimination, n.: Discrimination that happens when a system cannot see a trait like race directly but relies on something closely linked to it, such as a postcode, and so treats groups unequally anyway., RobodebtRobodebt, prop. n.: An Australian government program that used an automated calculation to accuse hundreds of thousands of welfare recipients of owing money, often wrongly. An official inquiry later found it unfair and illegal., sampling biassampling bias, n.: A distortion that arises when the data collected over-represents some kinds of people or situations and under-represents others, so conclusions drawn from it don't hold for everyone.