semantic textual similarity
In plain wordsHow close two sentences are in meaning, scored from 'about different things' to 'say exactly the same thing' — and the tests that check whether a computer's scores match people's judgments.
The task of predicting a graded score for how alike in meaning two short texts are, and the family of benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. built on it. In the SemEval shared tasks of 2012–2017, annotators rated sentence pairs from sources such as news headlines, image captions, forum posts and machine-translation output on a scale from 0 (different topics) to 5 (completely equivalent), and systems were scored by the Pearson or Spearman correlationcorrelation, n.: A measure of how closely two quantities move together, from −1 (one rises as the other falls) through 0 (no straight-line link) to +1 (they rise together). It does not show that one causes the other. between their predictions and the averaged human ratings. Unlike natural language inferencenatural language inference, n.: Deciding whether one sentence follows from another, contradicts it, or neither: if 'a man is playing a guitar on stage', is it true that 'a man is performing music'?, which asks whether one sentence entails another, similarity is symmetric and graded. The STS Benchmark (STS-B), about 8,600 English pairs selected from those tasks, became part of GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT., and STS tasks form one category of the Massive Text Embedding BenchmarkMTEB, prop. n.: A large public test suite that scores AI models which turn text into lists of numbers on many different jobs, such as searching, grouping, classifying and matching translations, so people can compare them on one leaderboard.. Since Sentence-BERTSentence-BERT, prop. n.: A way of turning a whole sentence into a list of numbers so that sentences with similar meanings get similar numbers, which makes it fast to search millions of texts by meaning rather than by exact words. (2019), STS has been a standard check on sentence embeddingstext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning., which are compared by cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.; a cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list. that reads both sentences together is usually more accurate but much slower.
The new embedding model gains two points of Spearman correlation on STS-B but does no better at retrieval.
Origin
Introduced as a pilot shared task at SemEval-2012 by Eneko Agirre, Daniel Cer, Mona Diab and Aitor Gonzalez-Agirre, building on earlier work on paraphrase identification. Annual tasks followed until 2017, when Cer and colleagues assembled the STS Benchmark; GLUE (2018) and MTEB (2022) carried it into general evaluation suites.
Usage note
Graded similarity is not agreement: a sentence and its negation share most of their content and may be rated fairly similar. High STS scores do not reliably predict how well embeddings perform in retrieval or clustering, one reason broader suites such as MTEB were built.
Related
- text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
- Sentence-BERTSentence-BERT, prop. n.: A way of turning a whole sentence into a list of numbers so that sentences with similar meanings get similar numbers, which makes it fast to search millions of texts by meaning rather than by exact words.
- cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.
- MTEBMTEB, prop. n.: A large public test suite that scores AI models which turn text into lists of numbers on many different jobs, such as searching, grouping, classifying and matching translations, so people can compare them on one leaderboard.
- GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT.
- semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'.
- cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list.
- natural language inferencenatural language inference, n.: Deciding whether one sentence follows from another, contradicts it, or neither: if 'a man is playing a guitar on stage', is it true that 'a man is performing music'?
Sources
- Agirre, E., Cer, D., Diab, M. & Gonzalez-Agirre, A. (2012). SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity. Proceedings of *SEM 2012.
- Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I. & Specia, L. (2017). SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Cross-lingual Focused Evaluation. Proceedings of SemEval-2017.