item response theory
In plain wordsA set of statistical models from educational testing that work out both how able each test-taker is and how hard each question is from who answered what correctly; now also used to evaluate AI models.
A family of statistical models that explain each person's response to each test item by a latent ability θ of the respondent and by parameters of the item. In the standard logistic models, the probability of a correct answer rises along an S-shaped item characteristic curve as θ exceeds the item's difficulty b; the curve's steepness is the item's discrimination a, and a floor c allows for lucky guesses (the one-, two- and three-parameter logistic models, the first equivalent to the Rasch model). Fitted jointly over many respondents and items, IRT places abilities and difficulties on a common scale, exposes uninformative or mislabeled items and underlies computerized adaptive testing. In AI evaluation, models play the test-takers: IRT is used to find which benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. items actually separate models, to estimate a model's full score from a small, well-chosen subset (as in tinyBenchmarks) and to link scores across benchmarks.
The IRT fit showed that a third of the benchmark's items were too easy to separate any of the frontier models.
Origin
Grew out of mid-20th-century psychometrics: Frederic Lord's 1952 monograph on test scores, Georg Rasch's 1960 model, and Allan Birnbaum's logistic models in Lord and Melvin Novick's 1968 textbook. The name 'item response theory' gradually displaced 'latent trait theory' and was entrenched by Lord's 1980 book. John Lalor, Hao Wu and Hong Yu were among the first to apply IRT to NLP evaluation, in 2016.
Usage note
Pairwise rating systems such as the Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths. behind arena leaderboards are close relatives: both posit a latent strength and a logistic link. IRT assumes a stable underlying trait; whether AI capability is one-dimensional enough for that is debated, and multidimensional IRT variants exist.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.
- Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
- LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
Sources
- Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum.
- Lalor, J. P., Wu, H. & Yu, H. (2016). Building an Evaluation Scale using Item Response Theory. Proc. EMNLP 2016.
- Polo, F. M. et al. (2024). tinyBenchmarks: Evaluating LLMs with Fewer Examples. Proc. ICML 2024.