independent component analysis
In plain wordsA technique for unmixing signals that have been blended together, such as separating individual voices from several microphones recording a crowded room.
A method that models observed multivariate data as linear mixtures of statistically independent, non-Gaussian source signals and recovers both the sources and the mixing matrix from the observations alone, a task called blind source separation. Whereas principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores. finds uncorrelated directions of maximal variance, ICA seeks components that are fully independent, which requires statistics beyond variance: algorithms maximize non-Gaussianity, measured by kurtosis or negentropy (as in FastICA), or maximize likelihood or minimize mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated. among the outputs (as in Infomax). Because mixtures tend to be more Gaussian than their sources (central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like.), maximizing non-Gaussianity recovers them; at most one source may be Gaussian, and the order and scale of the sources cannot be identified. ICA is used to separate audio sources, to remove eye-blink and muscle artifacts from EEG and fMRI recordings, and to analyze financial and image data; applied to natural images, it yields localized, oriented edge filters resembling receptive fields in the visual cortex.
ICA pulled the eye-blink artifacts out of the EEG as two clean components we could subtract.
Origin
Jeanny Hérault and Bernard Ans (from 1984) and Christian Jutten (1985–86) developed neural-network-like adaptive algorithms for blind source separation. Pierre Comon worked out its theory from 1991 and gave the formal definition of independent component analysis in 'Independent Component Analysis, a New Concept?' (1994). Anthony Bell and Terrence Sejnowski's Infomax algorithm (1995) and Aapo Hyvärinen and Erkki Oja's FastICA (1997–2000) made it practical and popular.
Related
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
- dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from.
- mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated.
- central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like.
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
Sources
- Comon, P. (1994). Independent Component Analysis, A New Concept? Signal Processing 36(3).
- Bell, A. J. & Sejnowski, T. J. (1995). An Information-Maximization Approach to Blind Separation and Blind Deconvolution. Neural Computation 7(6).
- Hyvärinen, A. & Oja, E. (2000). Independent Component Analysis: Algorithms and Applications. Neural Networks 13(4–5).