voice activity detection
In plain wordsDetecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk.
The classification of short audio frames, typically 10–30 milliseconds each, as speech or non-speech (silence, noise or music). It is a front end for many audio systems: it saves computation and bandwidth by passing only speech to speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video. or over a network, reduces transcription hallucinations during silence, supplies segments for speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are., and enables discontinuous transmission in mobile telephony, where generated comfort noise replaces silent periods. Classical detectors thresholded frame energy and zero-crossing rate or compared statistical models of speech and noise; telephone codec standards include their own VADs, and the open-source WebRTC VAD uses Gaussian mixture models. Small neural networks are now standard, as in the open-source Silero VAD and pyannote toolkits. In voice assistants and real-time voice agents, VAD drives endpointing (deciding that the user has finished an utterance) and 'barge-in' (letting the user interrupt the system's speech). Because pauses do not reliably mark the end of a turn, systems since 2024 increasingly add semantic turn-detection models that also weigh what was said.
The agent kept interrupting callers mid-sentence until we raised the VAD's end-of-speech timeout to 700 ms.
Origin
Speech detectors date to analog telephony: Bell Labs' Time Assignment Speech Interpolation (TASI, around 1960) reused transatlantic cable circuits during talkers' pauses. Digital mobile telephony standardized VADs in the 1990s, for example in GSM and in ITU-T codecs such as G.729 Annex B (1996).
Usage note
VAD answers 'is anyone speaking?', not 'who?' (speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are.) or 'what?' (speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.). A wake wordwake word, n.: A word or phrase, such as 'Alexa' or 'Hey Google', that a smart speaker or phone listens for constantly and that switches it into full listening mode. detector is a narrower trigger that listens for one particular phrase.
Related
- speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
- speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are.
- wake wordwake word, n.: A word or phrase, such as 'Alexa' or 'Hey Google', that a smart speaker or phone listens for constantly and that switches it into full listening mode.
- speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice.
- sound event detectionsound event detection, n.: Teaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops.
- latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
- wake wordwake word, n.: A word or phrase, such as 'Alexa' or 'Hey Google', that a smart speaker or phone listens for constantly and that switches it into full listening mode.
Sources
- Sohn, J., Kim, N. S. & Sung, W. (1999). A Statistical Model-Based Voice Activity Detection. IEEE Signal Processing Letters 6(1), 1–3.
- ITU-T (1996). Recommendation G.729 Annex B: A Silence Compression Scheme for G.729. International Telecommunication Union.
Mentioned in
speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are., speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice., voice agentvoice agent, n.: An AI system you talk to out loud, for example on a phone call, that listens, answers in a natural-sounding voice and can carry out tasks such as booking appointments or checking an order., wake wordwake word, n.: A word or phrase, such as 'Alexa' or 'Hey Google', that a smart speaker or phone listens for constantly and that switches it into full listening mode., WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio., sound event detectionsound event detection, n.: Teaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops., source separationsource separation, n.: Pulling apart a mixture of sounds into its separate parts, such as isolating one voice in a noisy party or the vocals, drums and bass in a finished song.