dimensionality reduction
In plain wordsSqueezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from.
A family of techniques for compressing the description of data: each item, originally measured on many variables, is mapped to a short list of coordinates, and methods differ in what the mapping must preserve, whether overall variance, pairwise distances, local neighborhoods or the separation between classes. One route keeps only some of the original variables (feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate.); the other computes new ones as combinations of them (feature extraction, also called projection). Linear projections include principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores., linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few. and non-negative matrix factorization; nonlinear methods include kernel PCA, manifold-learning techniques such as Isomap and locally linear embedding, neighbor-embedding methods for visualization (t-SNEt-SNE, n.: A technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data., UMAPUMAP, n.: A fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data.) and learned encoders such as autoencodersautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.. Reduction counters the curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much., strips noise and redundancy, cuts storage and computation, and makes structure visible in two or three dimensions. It works because of the manifold hypothesismanifold hypothesis, n.: The idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all.: real data recorded in many dimensions tend to lie near a much lower-dimensional surface.
After reducing the features to 50 principal components, clustering ran ten times faster with no loss in quality.
Origin
Its oldest workhorse, principal component analysis, goes back to Karl Pearson (1901) and Harold Hotelling (1933). Multidimensional scaling followed in psychometrics in the 1950s, and nonlinear manifold learning arrived with Isomap and locally linear embedding, published side by side in Science in December 2000. Neighbor-embedding methods (t-SNE in 2008, UMAP in 2018) made reduction a routine step for visualizing learned embeddings.
Usage note
Reduced dimensions are often hard to interpret, and nonlinear visualizations distort distances and densities; treat two-dimensional plots as exploratory rather than as evidence.
Related
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
- t-SNEt-SNE, n.: A technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data.
- UMAPUMAP, n.: A fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data.
- feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate.
- curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.
- manifold hypothesismanifold hypothesis, n.: The idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all.
- autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
Sources
- Tenenbaum, J. B., de Silva, V. & Langford, J. C. (2000). A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science 290(5500).
- Roweis, S. T. & Saul, L. K. (2000). Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science 290(5500).
Mentioned in
embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data., clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size., curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much., dimensionalitydimensionality, n.: How many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands., feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate., hierarchical clusteringhierarchical clustering, n.: A way of grouping data that builds a family tree of clusters, joining the most similar items step by step, so you can cut the tree at whatever level of detail you need., linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few., principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores., topic modeltopic model, n.: A method that reads a large pile of documents and discovers the themes running through them, describing each theme by its typical words and each document by its mix of themes., t-SNEt-SNE, n.: A technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data., UMAPUMAP, n.: A fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data., unsupervised learningunsupervised learning, n.: Machine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary., independent component analysisindependent component analysis, n.: A technique for unmixing signals that have been blended together, such as separating individual voices from several microphones recording a crowded room., Johnson–Lindenstrauss lemmaJohnson–Lindenstrauss lemma, n.: A surprising fact about high-dimensional data: a random projection can squash a large set of points into far fewer dimensions while distorting every distance between them only slightly., latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together., manifold hypothesismanifold hypothesis, n.: The idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all., self-organizing mapself-organizing map, n.: A neural network that arranges data on a flat grid so that similar items end up in neighboring cells, producing a map of how the data is organized., spectral clusteringspectral clustering, n.: A clustering method that treats data points as a network linked by similarity and finds natural groups by cutting the network where its links are weakest., Teuvo KohonenTeuvo Kohonen, prop. n.: A Finnish engineer who invented the self-organizing map, a neural network that arranges complex data on a simple grid so that similar items end up next to each other.