Skip to content
Last Dictionary
Plane
Evaluation & BenchmarksLanguage Models & Transformers

HELM

proper nounAlso: Holistic Evaluation of Language Models, HELM Lite, HELM Capabilities, HELM Safety

In plain wordsA large, open testing framework from Stanford that grades many language models on many kinds of tasks and qualities — not only accuracy but also fairness, toxicity and efficiency.

  1. A benchmark and evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically. from Stanford's Center for Research on Foundation Models (CRFM), updated continuously, that rates language models on many scenarios (task, domain, language) and many metrics simultaneously. Its first release ran 30 models on 42 scenarios (16 core, 26 targeted) and scored not only accuracy but also calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it., efficiency and three social dimensions — fairness, bias and toxicity — lifting the average coverage of core scenarios per model from 17.9% in earlier published evaluations to 96.0%. Every prompt and model output was made public for inspection. HELM has since grown into a family of leaderboards, including HELM Lite, HELM Capabilities and HELM Safety, and domain suites such as MedHELM for medicine and VHELM for vision-language models. Its emphasis on standardized conditions and multi-metric reporting shaped later evaluation practice.

    HELM shows the model is accurate but poorly calibrated on legal questions.

Origin

Introduced by Percy Liang, Rishi Bommasani, Tony Lee and some fifty colleagues at Stanford CRFM in November 2022, and published in Transactions on Machine Learning Research in 2023.

Usage note

HELM is both a methodology and software. Its leaderboards use different scenario sets, so scores from one are not interchangeable with another.

Related

See also
  • evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically.
  • benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
  • MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge.
  • BIG-benchBIG-bench, prop. n.: A huge collaborative test of 204 tasks, written by hundreds of researchers, built to probe what language models can and cannot do as they grow larger.

Sources

  1. Liang, P., Bommasani, R., Lee, T. et al. (2022). Holistic Evaluation of Language Models. arXiv:2211.09110; Transactions on Machine Learning Research (2023). arxiv.org/abs/2211.09110

Mentioned in

BIG-benchBIG-bench, prop. n.: A huge collaborative test of 204 tasks, written by hundreds of researchers, built to probe what language models can and cannot do as they grow larger., evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically., Stanford HAIStanford HAI, prop. n.: An artificial intelligence institute at Stanford University that funds research, advises policymakers and publishes the AI Index, a widely cited yearly report on the state of AI., Percy LiangPercy Liang, prop. n.: An American computer scientist at Stanford whose center coined the term 'foundation model' and builds tools for evaluating and documenting large AI models.