disentangled representation
In plain wordsAn internal representation in which each part tracks one separate, meaningful factor of the data, such as one number for a face's angle, another for its lighting and another for hair color.
A learned representation that sorts the independent sources of variation in the data, such as object identity, pose, lighting, color and position, into different dimensions or subspaces, so that moving along one latent coordinate alters a single factor while the rest stay fixed. Disentanglement is valued for interpretability, controllable generation, fairness (isolating a protected attribute) and sample-efficient transfer. Methods include the β-VAE, which up-weights the KL term of a variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. to push latents towards independence, FactorVAE, β-TCVAE and InfoGAN. Francesco Locatello and colleagues (2019) proved that unsupervised disentanglement is impossible without inductive biasesinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases. or some supervision, and found existing methods' scores to depend heavily on random seeds and hyperparameters. Interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. research pursues related goals with sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. and monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several..
In the β-VAE's disentangled representation, one latent controls rotation and another controls size.
Origin
Separating the underlying factors of variation in data is an older aim, pursued for example in Joshua Tenenbaum and William Freeman's bilinear models of style and content (2000). Yoshua Bengio, Aaron Courville and Pascal Vincent's 2012 review made disentangling those factors an explicit goal of deep representation learning. InfoGAN (Xi Chen and colleagues, 2016) and Irina Higgins and colleagues' β-VAE (ICLR 2017) offered practical methods; Locatello and colleagues' impossibility result won a best-paper award at ICML 2019.
Related
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
- variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
- latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
- monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.
Mentioned in
latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates., representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.