ROUGE
In plain wordsAn automatic score for summaries that checks how much of the wording in human-written reference summaries also appears in the computer-generated one.
A family of recall-oriented overlap scores, devised for judging automatic summaries and also applied to other generated text, that ask how much of the wording of one or more human-written references reappears in the candidate. ROUGE-N counts shared n-grams (ROUGE-1 for single words, ROUGE-2 for word pairs); ROUGE-L uses the longest common subsequence, rewarding words that appear in the same order without requiring them to be adjacent; ROUGE-S and ROUGE-SU count skip-bigrams. Implementations report recall, precision and their harmonic mean (F-measure), optionally after stemming. ROUGE became the standard metric for text summarizationtext summarization, n.: Using a computer to shorten a document, meeting or set of articles into a brief version that keeps the main points. benchmarks, but like BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations. it rewards surface overlap: it misses paraphrase, cannot detect factual errors or hallucinationshallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true., and correlates weakly with human judgments of abstractive summaries.
Our abstractive model scores lower on ROUGE-2 than the extractive baseline, yet raters prefer it.
Origin
Introduced by Chin-Yew Lin of the University of Southern California's Information Sciences Institute in 'ROUGE: A Package for Automatic Evaluation of Summaries', presented at the Text Summarization Branches Out workshop at ACL 2004 in Barcelona. The name deliberately echoes BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations.; the metric was adopted in NIST's Document Understanding Conference evaluations.
Usage note
Pronounced like the French word for 'red'. Implementations differ in tokenization and stemming and give different numbers, so reports should name the package used.
Related
- BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations.
- BERTScoreBERTScore, prop. n.: An automatic score for computer-written text that compares meanings rather than exact words, using an AI language model to judge how closely each word matches the words of a reference text.
- text summarizationtext summarization, n.: Using a computer to shorten a document, meeting or set of articles into a brief version that keeps the main points.
- human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
- LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
Sources
- Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out (ACL 2004 Workshop), Barcelona, 74–81. aclanthology.org/W04-1013
Mentioned in
BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations., human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well., natural language generationnatural language generation, n.: Getting computers to produce human language — from turning a spreadsheet into a weather report to writing an essay or a chatbot's reply., text summarizationtext summarization, n.: Using a computer to shorten a document, meeting or set of articles into a brief version that keeps the main points., BERTScoreBERTScore, prop. n.: An automatic score for computer-written text that compares meanings rather than exact words, using an AI language model to judge how closely each word matches the words of a reference text.