SQuAD
In plain wordsA Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016.
A machine-reading benchmark built by having crowdworkers read Wikipedia paragraphs and pose questions about them, each answerable by extracting a span from the paragraph. Version 1.1 holds over 100,000 question–answer pairs from more than 500 articles and is scored by exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters. and token-overlap F1F1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.; people reached 86.8% F1, far ahead of the authors' 51.0% logistic-regression baseline. Version 2.0 (2018) mixed in over 50,000 unanswerable questions, crafted by crowdworkers to look answerable, so a system must also recognize when the paragraph offers no answer and decline to respond — a change that dropped a strong neural model from 86% to 66% F1. Its public leaderboard turned machine reading into a fast-moving race: in January 2018 systems from Alibaba and Microsoft Research Asia exceeded the human exact-match score on SQuAD 1.1, and BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.-based models later surpassed humans on SQuAD 2.0. Critics noted that span selection rewards lexical matching more than understanding.
Extractive QA models fine-tuned on SQuAD struggle when the answer is not a contiguous span.
Origin
Introduced by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev and Percy Liang of Stanford University at EMNLP 2016. SQuAD 2.0 followed from Rajpurkar, Robin Jia and Liang at ACL 2018 in 'Know What You Don't Know'.
Usage note
Surpassing the human baseline was widely reported as machines 'reading better than humans'; the human figure, however, came from crowdworkers under task constraints. Robin Jia and Percy Liang showed in 2017 that adding distracting sentences sharply cut model accuracy.
Related
- question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know.
- exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters.
- F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
- abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
Sources
- Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016. arxiv.org/abs/1606.05250
- Rajpurkar, P., Jia, R. & Liang, P. (2018). Know What You Don't Know: Unanswerable Questions for SQuAD. ACL 2018. arxiv.org/abs/1806.03822
Mentioned in
exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good., question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know., Percy LiangPercy Liang, prop. n.: An American computer scientist at Stanford whose center coined the term 'foundation model' and builds tools for evaluating and documenting large AI models.