MMMU
In plain wordsA test for AI models that understand pictures as well as text: 11,500 college-level questions across 30 subjects, each built around one or more images such as a diagram, a musical score or a chemical structure.
A multimodal benchmark of about 11,500 questions drawn from college textbooks, quizzes and exams and organized into 30 subjects and 183 subfields under six broad disciplines: art and design; business; health and medicine; humanities and social science; science; and technology and engineering. Every question combines text with at least one image — 30 image types in all, from sheet music and chemical structures to maps, tables and diagrams — so a model must both read the picture correctly and apply expert subject knowledge; most items are multiple choice. At release GPT-4V scored 56%, and Google's Gemini Ultra reported 59% weeks later, both well below expert human performance; MMMU became the headline benchmark for multimodal modelsmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type. in launch reports. MMMU-Pro (2024) hardened it by filtering out questions answerable from text alone, adding answer options and embedding questions inside images.
The model gains five points on MMMU, mostly from better chart reading.
Origin
Introduced by Xiang Yue, Yuansheng Ni and colleagues, led from Ohio State University and the University of Waterloo, in November 2023, and presented at CVPR 2024. Its title frames it as a benchmark for 'expert AGI'.
Usage note
Some questions can be answered from the text without looking at the image, which inflates scores for language-only reasoning; MMMU-Pro was built to address this.
Related
- multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.
- MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
Sources
- Yue, X., Ni, Y. et al. (2024). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024. arxiv.org/abs/2311.16502