equivariance
In plain wordsA property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way.
The property of a function f that commutes with a group G of transformations: transforming the input by g ∈ G transforms the output by a corresponding action, f(g·x) = g·f(x). Convolutionalconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location. layers are equivariant to translations (shifting the image shifts the feature mapsfeature map, n.: The grid of numbers produced when one learned pattern detector is scanned across an image, showing where, and how strongly, that pattern appears.); graph neural networksgraph neural network, n.: A neural network for data shaped like a web of connections, such as molecules, social networks or road maps, in which each point repeatedly gathers information from its neighbors. are equivariant to relabeling of nodes; self-attention without positional encodingspositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model. is permutation-equivariant. It differs from invarianceinvariance, n.: A property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped., in which the output does not change at all; invariant outputs are usually obtained by pooling equivariant features. Group-equivariant networks extend the idea to rotations, reflections, scaling and 3-D rigid motions (E(3)- and SE(3)-equivariant networks), cutting data requirements in molecular modeling, physics and medical imaging. Building in known symmetries is the organizing principle of geometric deep learninggeometric deep learning, n.: An approach to designing neural networks around the shapes and symmetries of the data, whether grids, graphs, surfaces or 3-D structures, so the network respects how that data can be moved or rearranged..
Because the potential is rotation-equivariant, we no longer augment the molecules with random rotations.
Origin
A concept from group and representation theory. In deep learning it became an explicit design principle with Taco Cohen and Max Welling's 'Group Equivariant Convolutional Networks' (February 2016), building on earlier work on steerable filters and symmetry in vision; equivariant graph and point-cloud networks followed from about 2018.
Related
- invarianceinvariance, n.: A property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped.
- geometric deep learninggeometric deep learning, n.: An approach to designing neural networks around the shapes and symmetries of the data, whether grids, graphs, surfaces or 3-D structures, so the network respects how that data can be moved or rearranged.
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data.
- graph neural networkgraph neural network, n.: A neural network for data shaped like a web of connections, such as molecules, social networks or road maps, in which each point repeatedly gathers information from its neighbors.
- inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.
- capsule networkcapsule network, n.: A network design championed by Geoffrey Hinton in which groups of neurons, called capsules, each describe a part of an object and its position, so the network can check whether the parts fit together.
- invarianceinvariance, n.: A property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped.
Sources
- Cohen, T. S. & Welling, M. (2016). Group Equivariant Convolutional Networks. ICML 2016; arXiv:1602.07576. arxiv.org/abs/1602.07576
Mentioned in
convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects., data augmentationdata augmentation, n.: Creating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly., graph neural networkgraph neural network, n.: A neural network for data shaped like a web of connections, such as molecules, social networks or road maps, in which each point repeatedly gathers information from its neighbors., inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases., vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence., weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data., capsule networkcapsule network, n.: A network design championed by Geoffrey Hinton in which groups of neurons, called capsules, each describe a part of an object and its position, so the network can check whether the parts fit together., geometric deep learninggeometric deep learning, n.: An approach to designing neural networks around the shapes and symmetries of the data, whether grids, graphs, surfaces or 3-D structures, so the network respects how that data can be moved or rearranged., invarianceinvariance, n.: A property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped.