Skip to content
Last Dictionary
Plane
Language Models & TransformersHistory & LandmarksMathematics & Statistics

n-gram model

nounAlso: n-gram language model, n-gram, bigram model, trigram model, Kneser–Ney smoothing, backoff

In plain wordsAn early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text.

  1. A language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. that predicts the next word from how often short word sequences (n-grams) occurred in a training corpus, deliberately ignoring everything earlier than the last n − 1 words (a Markov assumption of order n − 1). Counting alone would declare impossible any sequence the corpus happens to lack, and most long sequences are absent from any corpus, so practical models hold back some probability for the unseen through smoothing and backoff: Katz backoff, interpolation and above all Kneser–Ney smoothing. Bigram (n = 2), trigram (n = 3) and higher-order models were the workhorse of speech recognition and statistical machine translation from the 1980s to the early 2010s; Google's 2006 Web 1T dataset counted sequences of up to five words over about a trillion words of web text. Neural models replaced them, but n-gram statistics remain useful for data filtering, contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability. checks and corpus analysis.

    A 5-gram model with Kneser–Ney smoothing was our baseline for the speech recognizer.

P(wt∣w1:t−1)≈P(wt∣wt−n+1:t−1),P^MLE(wt∣wt−n+1:t−1)=C(wt−n+1:t)C(wt−n+1:t−1)P(w_t \mid w_{1:t-1}) \approx P(w_t \mid w_{t-n+1:t-1}), \qquad \hat{P}_{\mathrm{MLE}}(w_t \mid w_{t-n+1:t-1}) = \frac{C(w_{t-n+1:t})}{C(w_{t-n+1:t-1})}

Origin

Andrey Markov analyzed letter sequences in Pushkin's 'Eugene Onegin' in 1913, and Claude Shannon generated text from letter and word n-gram statistics in 'A Mathematical Theory of Communication' (1948). Frederick Jelinek's group at IBM made n-gram models central to statistical speech recognition from the 1970s; Reinhard Kneser and Hermann Ney's smoothing (1995) became the standard.

Related

See also
  • language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.
  • Markov chainMarkov chain, n.: A chance-driven process that moves between states without memory: the odds of its next move are set by where it is, never by the path that led there. Forecasting tomorrow's weather from today's alone is a simple example.
  • perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.
  • next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
  • hidden Markov modelhidden Markov model, n.: A model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds.
  • information theoryinformation theory, n.: A branch of mathematics that treats information as surprise — the less predictable a message, the more it tells you — and works out how far data can be compressed and how reliably it can be sent.

Sources

  1. Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal 27.
  2. Kneser, R. & Ney, H. (1995). Improved backing-off for M-gram language modeling. ICASSP 1995.

Mentioned in

language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations., Claude ShannonClaude Shannon, prop. n.: An American engineer and mathematician who invented information theory, the mathematics of bits and communication, and wrote one of the first plans for a chess-playing computer., I. J. GoodI. J. Good, prop. n.: A British statistician who broke codes with Alan Turing and in 1965 argued that a machine smarter than us could design still smarter machines, setting off an 'intelligence explosion'., Markov chainMarkov chain, n.: A chance-driven process that moves between states without memory: the odds of its next move are set by where it is, never by the path that led there. Forecasting tomorrow's weather from today's alone is a simple example., perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll., Frederick JelinekFrederick Jelinek, prop. n.: A Czech-born American engineer who led IBM's speech-recognition research and made statistics learned from data, rather than hand-written grammar rules, the basis of language technology., Tomas MikolovTomas Mikolov, prop. n.: A Czech computer scientist who created word2vec, a fast way to turn words into lists of numbers that capture their meaning, and pioneered language models built from recurrent networks.