source separation
In plain wordsPulling apart a mixture of sounds into its separate parts, such as isolating one voice in a noisy party or the vocals, drums and bass in a finished song.
The recovery of individual source signals from recordings in which they are mixed, most commonly in audio: separating overlapping talkers (speech separation), isolating speech from noise (speech enhancement) or splitting music into stems such as vocals, drums and bass. 'Blind' separation assumes little about the sources or how they were mixed: independent component analysis (1990s) separates statistically independent sources when there are at least as many microphones as sources, and non-negative matrix factorization of spectrogramsspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns. (2000s) splits single-channel mixtures into recurring spectral patterns. Deep learning made single-microphone separation practical: networks predict a time–frequency mask for each source or, as in Conv-TasNet (2019), separate directly in the waveform domain, while permutation-invariant training resolves which output should correspond to which speaker. Music tools such as Spleeter and Demucs (from 2019) made stem separation a consumer feature. Quality is measured with signal-to-distortion ratios such as SI-SDR. Applications include hearing aids, meeting transcription, karaoke, remixing and the restoration of archival recordings.
Run source separation before transcription; the vocals are buried under the crowd noise.
Origin
The 'cocktail party problem' of attending to one voice among many was named by the cognitive scientist Colin Cherry in 1953. Blind source separation emerged in signal processing in the 1980s with work by Jeanny Hérault and Christian Jutten, and developed into independent component analysis in the 1990s.
Usage note
Machine-learning separation has been used in audio restoration, notably to isolate John Lennon's voice from a 1970s home demo for the Beatles' 2023 single 'Now and Then'.
Related
- spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns.
- speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are.
- speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
- voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk.
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
- music generationmusic generation, n.: Using AI to compose or produce music — from melodies written as notes to complete songs with vocals — often from a short text description like 'upbeat jazz with saxophone'.
Sources
- Cherry, E. C. (1953). Some Experiments on the Recognition of Speech, with One and with Two Ears. Journal of the Acoustical Society of America 25(5), 975–979.
- Luo, Y. & Mesgarani, N. (2019). Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 27(8), 1256–1266.