counterfactual

Causality and statistics

A claim or quantity about the outcome that would have occurred, for a particular unit or situation, had some antecedent been different from what actually happened. In the potential-outcomes framework every unit has an outcome under each treatment, only one of which is ever observed — the 'fundamental problem of causal inference' — so causal effects are contrasts involving counterfactual outcomes. In Judea Pearl's structural causal modelsstructural causal model, n.: A precise description of what causes what in a system — a diagram of arrows plus equations — that predicts what would happen if you intervened, not just what tends to occur together., counterfactuals are computed in three steps: abduction (inferring unobserved background conditions from the evidence), action (intervening to set the antecedent) and prediction. Pearl places them on the top rung of his 'ladder of causation', above association and intervention, because they require the most causal knowledge.

Machine learning

A minimally modified input used to probe or explain a model. Counterfactual explanationscounterfactual explanation, n.: An explanation that tells someone the smallest change that would have produced a different outcome — for example, that the application would have been accepted if it had listed two more years of work experience. state what would have to change for a decision to flip (a loan approved had income been higher); counterfactual data augmentation swaps attributes to weaken spurious correlationsspurious correlation, n.: A link in data that fools you: two things move together, yet neither drives the other, because a hidden shared cause, the way the data were collected or luck produced the pattern. AI models readily latch onto such links.; counterfactual fairnesscounterfactual fairness, n.: A fairness test that asks: had this person been of another race or sex, with everything that would have followed from that, would the system have decided the same way? asks whether a decision would change had a protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly. been different; and activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. runs counterfactual forward passes inside a network.

A counterfactual with only the applicant's postcode changed flipped the decision, a red flag for proxy discrimination.