t-SNE
In plain wordsA technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data.
A nonlinear dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from. method for visualization ('t-distributed stochastic neighbor embedding'). It turns pairwise distances in the original space into probabilities that points are neighbors, using Gaussian kernels whose widths are set per point to match a user-chosen perplexity, roughly the effective number of neighbors (typically 5–50). It then arranges the points in two or three dimensions so that a matching neighbor distribution, computed with a heavy-tailed Student t kernel, is as close as possible to the original, minimizing the Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. by gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.. The heavy tails let moderately distant points spread apart, relieving the crowding that afflicted its predecessor SNE and yielding well-separated clusters. Tree-based (Barnes–Hut) and interpolation-based approximations scale it to millions of points. It preserves local neighborhoods but not global geometry: cluster sizes and between-cluster distances in the plot carry little meaning, results vary with perplexity and random seed, and apparent clusters can emerge from unstructured data.
In the t-SNE plot the fraud cases formed a tight island, but we checked it at several perplexities before believing it.
Origin
Geoffrey Hinton and Sam Roweis presented stochastic neighbor embedding (SNE) at the NIPS conference in 2002. Laurens van der Maaten and Hinton replaced its low-dimensional Gaussian with a Student t-distribution in 'Visualizing Data using t-SNE' (Journal of Machine Learning Research, 2008). Van der Maaten's Barnes–Hut approximation (2014) made it practical for large datasets, and it became a standard tool in genomics and for inspecting neural-network embeddings.
Usage note
Read a t-SNE map for which points are neighbors, not for how far apart or how large clusters are. Running it on PCA-reduced data (say, 50 dimensions) is a common speed-up.
Related
- UMAPUMAP, n.: A fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data.
- dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from.
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
- clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size.
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
Sources
- van der Maaten, L. & Hinton, G. (2008). Visualizing Data using t-SNE. Journal of Machine Learning Research 9: 2579–2605. jmlr.org/papers/v9/vandermaaten08a.html
- Hinton, G. & Roweis, S. (2002). Stochastic Neighbor Embedding. Advances in Neural Information Processing Systems 15.
Mentioned in
embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers., dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., UMAPUMAP, n.: A fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data., manifold hypothesismanifold hypothesis, n.: The idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all., self-organizing mapself-organizing map, n.: A neural network that arranges data on a flat grid so that similar items end up in neighboring cells, producing a map of how the data is organized., Teuvo KohonenTeuvo Kohonen, prop. n.: A Finnish engineer who invented the self-organizing map, a neural network that arranges complex data on a simple grid so that similar items end up next to each other.