LMArena
In plain wordsA public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
A crowdsourced evaluation platform that ranks models by blind pairwise preference: a visitor types a prompt, two randomly chosen models answer side by side without names, and the visitor picks the better reply or declares a tie, learning which models took part only afterward. Votes are aggregated with a Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths. into scores on an EloElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one.-like scale with confidence intervals, shown on public leaderboards for text, vision, web development, search, image and video generation, and agent tasks. Because it uses real users' fresh prompts rather than a fixed test set, it is hard to contaminate and tracks what people value, and its rankings became a fixture of model launches. Criticisms include sensitivity to style — length, formatting and flattery — which the platform counters with 'style control' adjustments; the unrepresentativeness of its users; and the advantage developers gain by privately testing many variants and accessing battle data, documented in the 2025 paper 'The Leaderboard Illusion'. In April 2025 Meta was criticized for promoting the second-place ranking of an unreleased, chat-optimized Llama 4 variant that differed from the public model.
The model debuted first on LMArena's text leaderboard, though only six points ahead of the runner-up.
Origin
Launched in spring 2023 (announced 3 May) as Chatbot Arena by LMSYS, a student-led research group centered at UC Berkeley, with Wei-Lin Chiang, Anastasios Angelopoulos and Ion Stoica among its founders. It moved to lmarena.ai in September 2024, incorporated in April 2025, rebranded as Arena in January 2026, and raised a $200 million Series B at a $3.1 billion valuation in October 2026.
Usage note
The headword is the 2024–2025 name; since January 2026 the service has been called Arena. A score reflects the average preferences of the platform's users, not correctness on specialist tasks.
Related
- leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.
- Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one.
- Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.
- human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
- win ratewin rate, n.: The share of head-to-head comparisons in which judges — people or another AI — prefer one model's answer over a rival's for the same request.
- sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful.
- LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
Sources
- Chiang, W.-L., Zheng, L., Sheng, Y. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ICML 2024. arxiv.org/abs/2403.04132
- Singh, S., Nan, Y. et al. (2025). The Leaderboard Illusion. arXiv:2504.20879. arxiv.org/abs/2504.20879
Mentioned in
evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews., leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models., Meta AIMeta AI, prop. n.: The artificial intelligence research group of Meta, the company behind Facebook and Instagram, known for the PyTorch software and the Llama models. It is also the name of Meta's chatbot., confidence intervalconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample., Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one., human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well., Nano BananaNano Banana, prop. n.: The nickname, later Google's official branding, for its Gemini image models, a viral hit in 2025 for editing photos from simple instructions while keeping people looking like themselves., preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., vibe checkvibe check, n.: Trying out a new AI model informally — chatting with it, asking your favorite test questions — to get a gut feeling for how good it is, rather than measuring it with formal tests., win ratewin rate, n.: The share of head-to-head comparisons in which judges — people or another AI — prefer one model's answer over a rival's for the same request., benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks., Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths., item response theoryitem response theory, n.: A set of statistical models from educational testing that work out both how able each test-taker is and how hard each question is from who answered what correctly; now also used to evaluate AI models., MT-BenchMT-Bench, prop. n.: A set of 80 two-part questions — in writing, reasoning, math, coding and more — whose answers are graded by a strong AI model, used to compare chat assistants quickly.