distributional hypothesis
In plain wordsThe idea that words found in similar company — surrounded by similar words — tend to be similar in meaning, the insight that lets computers learn something about word meaning from large amounts of text alone.
The hypothesis, credited to Zellig Harris and J. R. Firth, that words found in similar linguistic contexts tend to be alike in meaning, so that a word's meaning can be approximated by statistics of the contexts in which it appears. It underlies distributional semantics, which represents words as vectors of co-occurrence counts or learned embeddingsword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'., as in latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together., word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text. and GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text., and more broadly the self-supervised learning of language modelslanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. from raw text. It captures similarity and relatedness well but blurs distinctions such as antonymy, since 'hot' and 'cold' share contexts. Linguists and philosophers debate how much of meaning — reference, truth, connection to perception and action — can be recovered from distribution alone, a question central to arguments over whether language models understand language (see symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words. and octopus testoctopus test, n.: A thought experiment in which a clever octopus learns to impersonate a stranded islander by eavesdropping on an undersea cable, without ever knowing what the words refer to. It asks whether text-trained AI can grasp meaning.).
Embedding methods are the distributional hypothesis operationalized: count or predict contexts, then compare words by their vectors.
Origin
Usually credited to the linguist Zellig Harris ('Distributional Structure', 1954) and to J. R. Firth, whose 1957 remark about knowing a word by the company it keeps became its slogan; related ideas appear in Ludwig Wittgenstein's account of meaning as use.
Usage note
Weak versions (distribution correlates with meaning) are uncontroversial; strong versions (meaning just is distribution) are disputed.
Related
- word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
- word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text.
- GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text.
- latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together.
- symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words.
- octopus testoctopus test, n.: A thought experiment in which a clever octopus learns to impersonate a stranded islander by eavesdropping on an undersea cable, without ever knowing what the words refer to. It asks whether text-trained AI can grasp meaning.
- language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- bag of wordsbag of words, n.: A simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list.
Sources
- Harris, Z. S. (1954). Distributional Structure. Word 10(2–3).
- Firth, J. R. (1957). A synopsis of linguistic theory, 1930–1955. In Studies in Linguistic Analysis. Oxford: Blackwell.
Mentioned in
bag of wordsbag of words, n.: A simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list., computational linguisticscomputational linguistics, n.: The science of human language studied with computers: using computational models to understand how language works, and building the language technology that grows out of that understanding., symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words., octopus testoctopus test, n.: A thought experiment in which a clever octopus learns to impersonate a stranded islander by eavesdropping on an undersea cable, without ever knowing what the words refer to. It asks whether text-trained AI can grasp meaning., word sense disambiguationword sense disambiguation, n.: Working out which meaning of a word is intended in context — whether 'bank' means a riverside or a place that keeps money.