HellaSwag
In plain wordsA test in which an AI must pick the most sensible ending for a description of an everyday scene; the wrong endings were machine-written to fool computers, so people found it easy and early AI did not.
A commonsense inference benchmark in which a model chooses the most plausible continuation of a short description of an everyday activity — drawn from ActivityNet video captions and WikiHow articles — from four options. Its three wrong endings are machine-generated and were picked by Adversarial Filtering, in which successive rounds of discriminator models keep only the generated endings they find hardest to tell apart from the real one; human annotators then confirmed the correct choice. The result was easy for people, who scored above 95%, but not for the best models of 2019, fine-tuned BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. among them, which stayed under 48% — a sign that strong scores on the predecessor dataset SWAG had leaned on statistical artifacts. Large language models closed the gap within about four years — GPT-4GPT-4, prop. n.: OpenAI's 2023 language model, a big step up from the one behind the first ChatGPT. It scored well on many professional and academic exams and could also read images. reported 95.3% in 2023 — and HellaSwag became a routine, near-saturated entry in model evaluations, useful mainly for detecting regressions.
HellaSwag is near ceiling, so we mainly use it to catch regressions after quantization.
Origin
Introduced by Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi (University of Washington and the Allen Institute for AI) at ACL 2019, as a harder successor to their 2018 SWAG dataset. The name puns on the slang intensifier 'hella', though the paper also expands it as a backronym.
Usage note
Later audits reported ungrammatical and ambiguous items, so differences of a point or two at high accuracy are not meaningful.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- commonsense reasoningcommonsense reasoning, n.: The everyday background knowledge and reasoning people take for granted — dropped glasses break, hungry people eat, nobody is in two places at once — which machines long struggled to match.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- Winograd schema challengeWinograd schema challenge, prop. n.: A test of machine common sense built from pairs of sentences in which changing one word changes what a pronoun refers to — something people resolve instantly using everyday knowledge.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
Sources
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A. & Choi, Y. (2019). HellaSwag: Can a Machine Really Finish Your Sentence? ACL 2019. arxiv.org/abs/1905.07830