BIG-bench
In plain wordsA huge collaborative test of 204 tasks, written by hundreds of researchers, built to probe what language models can and cannot do as they grow larger.
A crowdsourced benchmark whose 204 tasks, submitted by researchers from 132 institutions (450 authors in all), target abilities the language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. of the day were expected to lack, so that gains with scale could be charted. Subjects run from linguistics and arithmetic to code, science, social bias and common sense; most tasks are JSON files scored by exact answers or multiple choice, the rest small programs. Its analysis is best known for finding that abrupt 'breakthrough' jumps at a particular scale tended to occur on multi-step tasks or under all-or-nothing metrics, while steady gains came mostly on knowledge-heavy tasks — evidence central to debates over emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured.; overall, bigger models scored higher and were better calibratedcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time. but remained weak in absolute terms. BIG-Bench Hard (BBH), a 23-task subset on which models had trailed average human raters, became a standard test of chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. prompting. Task files carry a canary stringcanary string, n.: A unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization. to help keep them out of training data.
On BIG-Bench Hard, chain-of-thought prompting lifts the model above the average human rater.
Origin
Organized from 2021 by researchers at Google with outside collaborators; the paper 'Beyond the Imitation Game', led by Aarohi Srivastava, appeared on arXiv in June 2022 and in Transactions on Machine Learning Research in 2023. Mirac Suzgun and colleagues introduced BIG-Bench Hard in October 2022.
Usage note
The name alludes to Alan Turing's 'imitation game', the original form of the Turing testTuring test, n.: Alan Turing's proposed test of machine intelligence: a judge holds typed conversations with a hidden person and a hidden computer, and the computer passes if the judge cannot consistently pick it out.. Task quality varies because tasks were volunteered, so most later work reports BBH or BIG-Bench Lite rather than the full suite.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- canary stringcanary string, n.: A unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization.
- HELMHELM, prop. n.: A large, open testing framework from Stanford that grades many language models on many kinds of tasks and qualities — not only accuracy but also fairness, toxicity and efficiency.
- Turing testTuring test, n.: Alan Turing's proposed test of machine intelligence: a judge holds typed conversations with a hidden person and a hidden computer, and the computer passes if the judge cannot consistently pick it out.
Sources
- Srivastava, A. et al. (2022). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615; TMLR (2023). arxiv.org/abs/2206.04615
- Suzgun, M. et al. (2023). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. Findings of ACL 2023.
Mentioned in
emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured., HELMHELM, prop. n.: A large, open testing framework from Stanford that grades many language models on many kinds of tasks and qualities — not only accuracy but also fairness, toxicity and efficiency., canary stringcanary string, n.: A unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization.