Skip to content
Last Dictionary
Plane
Evaluation & BenchmarksLanguage Models & Transformers

MT-Bench

proper noun

In plain wordsA set of 80 two-part questions — in writing, reasoning, math, coding and more — whose answers are graded by a strong AI model, used to compare chat assistants quickly.

  1. A benchmark of 80 open-ended questions, each followed by a second-turn request, that tests how well chat assistants follow instructions and cope with follow-ups across eight categories: coding, math, reasoning, extraction, writing, roleplay, and knowledge of STEM and of the humanities and social sciences. Answers are graded by an LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers., originally GPT-4, either on a 1–10 scale or by pairwise comparison with another model; for math and reasoning questions the judge is given a reference answer. Its authors showed that GPT-4's judgments agreed with expert human preferences over 80% of the time, about as often as humans agree with one another, which helped legitimize model-graded evaluation. MT-Bench was widely reported for open chat modelschat model, n.: A language model trained to hold a back-and-forth conversation, keeping track of who said what and replying as a helpful assistant. It is the engine inside chat apps. in 2023–2024, but with so few questions and a judge that favors certain styles it saturated quickly and was superseded by larger sets such as Arena-Hard.

    Its MT-Bench score of 8.9 sits close to the judge model's own, so we stopped relying on it.

Origin

Introduced by Lianmin Zheng, Wei-Lin Chiang, Ying Sheng and colleagues at LMSYS in June 2023, in the paper that popularized the phrase 'LLM-as-a-judge' and described Chatbot Arena (NeurIPS 2023 Datasets and Benchmarks Track).

Usage note

'MT' stands for multi-turn, not machine translation.

Related

See also
  • LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
  • LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
  • win ratewin rate, n.: The share of head-to-head comparisons in which judges — people or another AI — prefer one model's answer over a rival's for the same request.
  • human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
  • instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related.
  • chat modelchat model, n.: A language model trained to hold a back-and-forth conversation, keeping track of who said what and replying as a helpful assistant. It is the engine inside chat apps.

Sources

  1. Zheng, L., Chiang, W.-L., Sheng, Y. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track. arxiv.org/abs/2306.05685

Mentioned in

LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.