benchmaxxing
In plain wordsSlang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks.
Optimizing a model, or the way it is evaluated, to maximize published benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. scores rather than general capability. Practices range from legitimate but narrow (heavy training on benchmark-like tasks, tuning prompts and harnessesharness, n.: The software around an AI model that gives it instructions and tools, carries out its actions and decides when it stops. The model is the engine; the harness is the rest of the car. for each test) through questionable (selective reporting, privately testing many variants and publishing the best, submitting specially tuned versions to leaderboardsleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.) to improper (training on test datadata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.). The result is a gap between headline numbers and real-world usefulness — an instance of Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.. Calling a release 'benchmaxxed' became a common accusation in 2025 when scores outran users' vibe checksvibe check, n.: Trying out a new AI model informally — chatting with it, asking your favorite test questions — to get a gut feeling for how good it is, rather than measuring it with formal tests., and the practice fuels demand for private, rotating and contamination-resistant benchmarkscontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made..
It tops three leaderboards and still can't follow a two-step instruction — classic benchmaxxing.
Origin
Formed from 'benchmark' and the internet suffix '-maxxing' (as in 'looksmaxxing'), meaning to maximize one attribute obsessively. It spread on AI social media during 2025, a year of disputes such as the April 2025 controversy over a specially tuned version of Meta's Llama 4 entered on the LMArena leaderboard; Andrej Karpathy used it in his '2025 LLM Year in Review' (19 December 2025). Who first used it is not established.
Related
- Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
- vibe checkvibe check, n.: Trying out a new AI model informally — chatting with it, asking your favorite test questions — to get a gut feeling for how good it is, rather than measuring it with formal tests.
- LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
- leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.
- contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made.
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure., state of the artstate of the art, n.: The best performance or most advanced technique currently known in a field. In AI papers, a state-of-the-art result is one that beats every earlier published result on a test., vibe checkvibe check, n.: Trying out a new AI model informally — chatting with it, asking your favorite test questions — to get a gut feeling for how good it is, rather than measuring it with formal tests., nerfednerfed, adj.: Gaming slang for something made weaker by an update; applied to AI models that users believe got worse after a provider changed them, often without saying so., pelican on a bicyclepelican on a bicycle, n.: A joke test in which an AI model is asked to draw, by writing code, a pelican riding a bicycle; comparing the results became a popular, lighthearted way to judge each new model., tokenmaxxingtokenmaxxing, n.: Using as much AI as possible — huge prompts, long sessions, many agents at once — mainly to rank high on workplace measures of how much AI people use.