Simpson's paradox

A reversal, or vanishing, of an association when data from several subgroups are merged: the comparison that holds inside every subgroup fails for the pooled data, because the variable defining the subgroups is related both to the factor being compared and to the outcome. In the classic 1973 Berkeley graduate-admissions data, men were admitted at a higher overall rate than women, yet most departments admitted women at equal or higher rates: women had applied disproportionately to more selective departments. Which comparison is right cannot be settled from the data alone; it depends on the causal structure, and causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time. tools such as causal diagramsstructural causal model, n.: A precise description of what causes what in a system — a diagram of arrows plus equations — that predicts what would happen if you intervened, not just what tends to occur together. and the back-door criterion of the do-calculusdo-calculus, n.: A set of mathematical rules, devised by Judea Pearl, for working out the effect of deliberately changing something using only observational data plus a diagram of what causes what. say whether to condition on the grouping variable. In AI it matters for algorithmic fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not. audits, for comparing models evaluated on different mixes of task categories, and for A/B testingA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better. when the traffic mix differs between arms.

Model B had the higher win rate in every category yet the lower overall rate, because it was tested mostly on the hardest category.