testing with concept activation vectors
In plain wordsA method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect.
A concept-based global explanationglobal explanation, n.: An explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision. method that asks whether a concept people care about, defined only by a set of example images, pushes a network's predictions for a class up or down. A concept activation vector (CAV) is obtained by collecting a layer's activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. for example images of the concept (striped textures, in the original paper's example) and for random images, training a linear classifierlinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly. to separate them, and taking the vector normal to its decision boundary. For each input of a class (there, zebra), the directional derivative of the class logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. along the CAV shows whether moving activations toward the concept raises the prediction; the TCAV score is the fraction of class inputs for which it does. Statistical tests against CAVs trained on many random image sets guard against meaningless directions. TCAV needs no retraining, lets domain experts pose their own concepts, and was an early demonstration that human concepts correspond to linear directions in activation space — an idea central to later linear-representationlinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction. and steering work.
TCAV showed the skin-lesion classifier was highly sensitive to the 'ruler present' concept.
Origin
Introduced by Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viégas and Rory Sayres of Google in 'Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)' (arXiv, November 2017; ICML 2018). Automated concept discovery (ACE) followed in 2019.
Usage note
Results depend on how well the example images capture the concept and on the layer chosen; correlated concepts, such as stripes and savanna backgrounds, can be hard to separate. A CAV is a direction found by probing, the conceptual ancestor of today's steering vectorssteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it..
Related
- concept bottleneck modelconcept bottleneck model, n.: A model made to first predict human-understandable concepts — such as 'bone spur present' on an X-ray — and then decide only from those concepts, so people can inspect and correct them.
- linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
- steering vectorsteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.
- global explanationglobal explanation, n.: An explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision.
- linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
- network dissectionnetwork dissection, n.: A method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people.
Sources
- Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viégas, F. & Sayres, R. (2018). Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). Proceedings of ICML 2018.
Mentioned in
explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., global explanationglobal explanation, n.: An explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere., activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., concept bottleneck modelconcept bottleneck model, n.: A model made to first predict human-understandable concepts — such as 'bone spur present' on an X-ray — and then decide only from those concepts, so people can inspect and correct them., network dissectionnetwork dissection, n.: A method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people.