Elo rating
In plain wordsA number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one.
A rating system that estimates competitors' relative strength from pairwise outcomes. Each competitor has a rating, and the difference between two ratings predicts the expected score of a match through a logistic curve (formula below): a 400-point advantage implies roughly 10-to-1 odds. After each game a rating moves by K × (actual score − expected score), where the K-factor sets the size of updates. In AI, Elo-style ratings rank game-playing programs such as chess engines and AlphaGoAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it., agents trained by self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples., competitive-programming performance ('Codeforces Elo') and language models compared by human votes on LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.. Because sequential updates depend on the order of games, model leaderboards now usually fit a Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths. to all comparisons at once, as Chatbot Arena has done since December 2023, and report the result on an Elo-like scale.
A 100-point Elo gap means the stronger model should win about 64% of head-to-head votes.
Origin
Devised by Arpad Elo, a Hungarian-American physics professor and chess master. The United States Chess Federation adopted his system in 1960 and FIDE in 1970, and Elo described it in The Rating of Chessplayers, Past and Present (1978). LMSYS's Chatbot Arena brought Elo ratings to language-model evaluation in May 2023.
Usage note
Elo is a surname, not an acronym, so 'ELO' is wrong. Ratings are relative to a pool of competitors: numbers from different leaderboards or periods cannot be compared directly.
Related
- Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.
- LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
- leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.
- win ratewin rate, n.: The share of head-to-head comparisons in which judges — people or another AI — prefer one model's answer over a rival's for the same request.
- self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.
- AlphaGoAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it.
Sources
- Elo, A. E. (1978). The Rating of Chessplayers, Past and Present. Arco Publishing.
Mentioned in
leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models., LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings., win ratewin rate, n.: The share of head-to-head comparisons in which judges — people or another AI — prefer one model's answer over a rival's for the same request., AI co-scientistAI co-scientist, n.: An AI system built to work alongside human scientists, proposing and debating new research ideas and hypotheses that people then test in the lab., Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths., item response theoryitem response theory, n.: A set of statistical models from educational testing that work out both how able each test-taker is and how hard each question is from who answered what correctly; now also used to evaluate AI models.