word embedding
In plain wordsA list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
A dense real-valued vector, typically 50–1,000 dimensions, representing a word so that geometric relations among vectors mirror relations among words: words used in similar contexts receive nearby vectors (measured by cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.), and some relations appear as roughly constant offsets, as in the well-known king − man + woman ≈ queen analogy. Word embeddings rest on the distributional hypothesis that meaning shows in usage and are learned either by prediction (word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text., fastText) or by factorizing co-occurrence statistics (GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text., latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together.). Classic word embeddings are static — one vector per word form whatever the context — so they conflate senses such as 'bank'; contextual representations from ELMoELMo, prop. n.: A 2018 model whose word representations changed with context, so 'bank' got different numbers in 'river bank' and 'bank loan'. It was a stepping stone between fixed word vectors and BERT. (2018) and BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. superseded them for most tasks. They also absorb social biases present in their training text.
We initialized the classifier with 300-dimensional pretrained word embeddings.
Origin
The distributional idea is usually traced to Zellig Harris (1954) and to J. R. Firth's 1957 line 'You shall know a word by the company it keeps'. Yoshua Bengio and colleagues learned word vectors jointly with a neural language model (NIPS 2000; JMLR 2003); Ronan Collobert and Jason Weston (2008) and word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text. (2013) made pretrained embeddings standard inputs to NLP systems.
Usage note
Analogy arithmetic works less reliably than often claimed, partly because standard evaluations exclude the input words from the candidate answers.
Related
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text.
- GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text.
- cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.
- latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
- text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
- text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
Sources
- Bengio, Y., Ducharme, R., Vincent, P. & Jauvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research 3.
- Peters, M. E. et al. (2018). Deep contextualized word representations. NAACL 2018. arxiv.org/abs/1802.05365
Mentioned in
embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., AdaGradAdaGrad, prop. n.: A training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones., bag of wordsbag of words, n.: A simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list., text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning., word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text., WordNetWordNet, prop. n.: A hand-made map of English vocabulary in which words sharing a meaning are bundled together and the bundles are linked — recording, say, that a dog is a kind of animal and a wheel is part of a car., Yoshua BengioYoshua Bengio, prop. n.: A Canadian computer scientist who helped found deep learning, including ideas behind today's language models, and who has become a leading voice on the risks of advanced AI., zero-shot learningzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training., distributional hypothesisdistributional hypothesis, n.: The idea that words found in similar company — surrounded by similar words — tend to be similar in meaning, the insight that lets computers learn something about word meaning from large amounts of text alone., ELMoELMo, prop. n.: A 2018 model whose word representations changed with context, so 'bank' got different numbers in 'river bank' and 'bank loan'. It was a stepping stone between fixed word vectors and BERT., EMNLPEMNLP, prop. n.: A leading yearly conference on getting computers to handle human language, founded in the 1990s for data-driven methods and now dominated by work on large language models., GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text., iterative nullspace projectioniterative nullspace projection, n.: An early method for removing a trait such as gender from a model's internal representations by repeatedly training a simple detector for it and then flattening away whatever the detector relied on., latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together., Tomas MikolovTomas Mikolov, prop. n.: A Czech computer scientist who created word2vec, a fast way to turn words into lists of numbers that capture their meaning, and pioneered language models built from recurrent networks., vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's., word sense disambiguationword sense disambiguation, n.: Working out which meaning of a word is intended in context — whether 'bank' means a riverside or a place that keeps money.