Skip to content
Last Dictionary
Plane
Vision, Speech & PerceptionDeep Learning

sound event detection

nounAlso: SED, audio event detection, acoustic event detection, audio tagging, acoustic scene classification, AudioSet, DCASE

In plain wordsTeaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops.

  1. The task of recognizing which sound events occur in an audio recording, such as speech, a dog bark, a car horn or breaking glass, and estimating the onset and offset time of each occurrence. Polyphonic sound event detection handles overlapping events, the normal case in real environments. Related tasks are audio tagging (clip-level labels without timing), acoustic scene classification (recognizing the setting, such as a street, park or metro station), and sound event localization and detection, which also estimates each sound's direction. Typical systems compute log-mel spectrogramsmel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio. and apply convolutional–recurrent networks or audio transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. pretrained on large tagged collections, chiefly Google's AudioSet (2017), about two million ten-second YouTube clips labeled with 527 classes drawn from an ontology of 632 sound categories. Because precise timestamps are expensive to annotate, many systems learn from weak, clip-level labels. Evaluation uses event- or segment-based F1 scores and the polyphonic sound detection score. Applications include smart-home and security alerts, hearing aids, bioacoustic monitoring of wildlife, machine-fault detection and audio search.

    The detector flags breaking glass within 200 ms but confuses some alerts with cutlery dropped on tiles.

Origin

Acoustic event detection was evaluated in the CLEAR workshops of 2006–2007 on meeting-room audio. The DCASE challenges (Detection and Classification of Acoustic Scenes and Events), first run in 2013 and annual since 2016, and Jort Gemmeke and colleagues' AudioSet (ICASSP 2017) established the modern field.

Usage note

Distinct from speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., which transcribes words, and from voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk., which only separates speech from everything else.

Related

See also
  • spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns.
  • mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio.
  • voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk.
  • audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words.
  • multilabel classificationmultilabel classification, n.: Tagging each item with any number of labels at once — a photo can show a beach, a dog and a sunset — instead of picking just one category.
  • weak supervisionweak supervision, n.: Training AI with labels that are cheap but imperfect, such as rough rules of thumb, partial tags or noisy guesses, instead of carefully hand-checked answers.

Sources

  1. Gemmeke, J. F. et al. (2017). Audio Set: An Ontology and Human-Labeled Dataset for Audio Events. ICASSP 2017.
  2. Mesaros, A., Heittola, T., Virtanen, T. & Plumbley, M. D. (2021). Sound Event Detection: A Tutorial. IEEE Signal Processing Magazine 38(5), 67–83.

Mentioned in

audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., mel spectrogrammel spectrogram, n.: A version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio., speaker diarizationspeaker diarization, n.: Working out who spoke when in a recording, splitting a meeting or phone call into turns labeled Speaker 1, Speaker 2 and so on, without necessarily knowing who the speakers are., spectrogramspectrogram, n.: A picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns., voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk.