wav2vec

A family of self-supervisedself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image. models that learn speech representations directly from raw audio waveforms, best known through wav2vec 2.0. A convolutionalconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. feature encoder turns the waveform into a latent vector about every 20 milliseconds; spans of these are hidden, and a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. context network must identify, from a set of distractors, the quantized form of each hidden latent — a contrastivecontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things. objective in which the codebook itself is learned. The pretrained model is then fine-tunedfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. on transcribed audio with a CTCconnectionist temporal classification, n.: A training method that lets a network learn to transcribe speech or handwriting without being told exactly where each letter or sound begins and ends. loss. Its headline result concerned scarce labels: pretrained on 53,000 hours of untranscribed audio and fine-tuned on just ten minutes of transcribed speech, it scored 4.8% and 8.2% word error rateword error rate, n.: The standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better. on the clean and other LibriSpeechLibriSpeech, prop. n.: A free collection of about 1,000 hours of volunteers reading English audiobooks, matched with the book text; it became the standard yardstick for speech-recognition systems. test sets, against 1.8% and 3.3% with all 960 labeled hours — evidence that recognizers could be built for languages with little transcribed speech. Multilingual versions (XLS-R, 2021; Massively Multilingual Speech, 2023) and relatives such as HuBERT (2021), which predicts cluster labels rather than contrasting, followed.

We fine-tuned a multilingual wav2vec checkpoint on four hours of transcribed Quechua.