manifold hypothesis
In plain wordsThe idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all.
The conjecture that the inputs a learner actually meets — natural images, sounds, embeddings of text — fill almost none of their nominal high-dimensional space, clustering instead on or near one or a few smooth low-dimensional surfaces (manifolds) whose own coordinates capture the meaningful variation. Almost all random pixel arrays look like noise, so natural images occupy a vanishing fraction of pixel space, and a few factors of variation such as pose, lighting and identity move along the manifold. The hypothesis explains how learning escapes the curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.: sample needs depend on the data's intrinsic dimensionalitydimensionality, n.: How many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands. rather than the number of raw features. It motivates manifold-learning methods (Isomap, locally linear embedding, t-SNEt-SNE, n.: A technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data., UMAPUMAP, n.: A fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data.), the view of deep networks as flattening and untangling data manifolds, latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates. interpolation in autoencodersautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data. and generative modelsgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., and accounts of adversarial examplesadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon. as points pushed slightly off the manifold. It is supported empirically, for example by low intrinsic-dimension estimates for image datasets, but is not a theorem.
If the manifold hypothesis holds, interpolating in latent space should yield realistic faces, and it mostly does.
Origin
Bernhard Riemann introduced the manifold (Mannigfaltigkeit) in his 1854 habilitation lecture, published posthumously in 1868. Isomap (Joshua Tenenbaum, Vin de Silva and John Langford) and locally linear embedding (Sam Roweis and Lawrence Saul), both published in Science in 2000, launched manifold learning; the phrase 'manifold hypothesis' was in use by 2010, when Hariharan Narayanan and Sanjoy Mitter analyzed how to test it, followed by Charles Fefferman, Mitter and Narayanan (2016).
Related
- dimensionalitydimensionality, n.: How many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands.
- curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.
- latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
- dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
- UMAPUMAP, n.: A fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data.
Mentioned in
curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much., dimensionalitydimensionality, n.: How many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands., dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates., semi-supervised learningsemi-supervised learning, n.: Machine learning that combines a small set of labeled examples with a much larger pile of unlabeled ones, which are cheaper to collect, to learn better than from the labeled data alone., UMAPUMAP, n.: A fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data.