word error rate
In plain wordsThe standard score for speech recognition: the share of words a system gets wrong compared with a correct transcript, counting wrong, missing and extra words. Lower is better.
The standard accuracy measure for speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video. and other transcription systems: the number of word substitutions (S), deletions (D) and insertions (I) needed to turn the system's output into the reference transcript, divided by the number of words in the reference (N). The alignment is found by minimum edit (Levenshtein) distance at the word level, computed with dynamic programming. Because insertions are counted, WER can exceed 100%. Results depend heavily on text normalization (casing, punctuation, numbers, spelling variants), so systems should be compared on identically normalized text; for languages written without spaces between words, such as Chinese and Japanese, character error rate (CER) is used instead. WER weights all errors equally, so dropping 'not' counts the same as misspelling a name, and it ignores whether the meaning survived; complementary measures include accuracy on named entities, semantic metrics and task success. Top systems report WERs below 2% on read-speech benchmarks such as LibriSpeechLibriSpeech, prop. n.: A free collection of about 1,000 hours of volunteers reading English audiobooks, matched with the book text; it became the standard yardstick for speech-recognition systems. test-clean, but much higher figures on spontaneous, accented, noisy or overlapping speech.
The new model cut WER on our call-center set from 14% to 9%, mostly by fixing product names.
Origin
Word-level error rates computed by dynamic-programming alignment were in standard use in US government (DARPA and NIST) speech-recognition benchmark evaluations by the late 1980s, whose scoring tools fixed the modern definition; earliest use of the phrase has not been traced. The underlying edit distance was defined by Vladimir Levenshtein in 1965.
Usage note
Word accuracy is sometimes reported as 1 − WER. Because a single wrong word can reverse a meaning, a low WER does not guarantee transcripts fit for medical or legal use.
Related
- speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
- LibriSpeechLibriSpeech, prop. n.: A free collection of about 1,000 hours of volunteers reading English audiobooks, matched with the book text; it became the standard yardstick for speech-recognition systems.
- WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- optical character recognitionoptical character recognition, n.: Software that picks out the letters and words in a picture — a road sign, a handwritten note, a page of an old book — and turns them into real text a computer can copy, search or read aloud.
Sources
- Levenshtein, V. I. (1966). Binary Codes Capable of Correcting Deletions, Insertions, and Reversals. Soviet Physics Doklady 10(8), 707–710.
Mentioned in
optical character recognitionoptical character recognition, n.: Software that picks out the letters and words in a picture — a road sign, a handwritten note, a page of an old book — and turns them into real text a computer can copy, search or read aloud., speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., WhisperWhisper, prop. n.: A free OpenAI model that writes down what is said in recordings in many languages, and can translate it into English; it learned from hundreds of thousands of hours of online audio., ConformerConformer, prop. n.: A neural-network design for speech recognition that combines a transformer, which relates distant parts of a recording, with convolutions, which pick up fine local sound patterns., LibriSpeechLibriSpeech, prop. n.: A free collection of about 1,000 hours of volunteers reading English audiobooks, matched with the book text; it became the standard yardstick for speech-recognition systems., wav2vecwav2vec, prop. n.: A family of Meta speech models that learn the sounds of language from thousands of hours of unlabeled recordings, so that only minutes of transcribed speech are needed to build a recognizer.