Potemkin understanding
In plain wordsWhen an AI can correctly explain a concept yet cannot use it properly, revealing that its apparent understanding is a facade — like the fake village fronts said to have been built to impress a Russian empress.
A failure mode, proposed in a 2025 study of large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., in which a model answers the questions that would show understanding of a concept in a person — typically by giving a correct definition — but then applies the concept in ways no person who could give that definition would, for instance explaining a rhyme scheme correctly and then writing a poem that breaks it. Its authors' point is that a test built for people licenses conclusions about a model only if the model errs the way people do; otherwise a high score may certify a facade. With tests that ask a model both to explain a concept and to use it, they found such 'potemkins' pervasive in the models and domains they examined, and traced them to internally inconsistent concept representations rather than simply to missing knowledge.
It aced the definitions quiz, then misapplied every definition — textbook Potemkin understanding.
Origin
Introduced by Marina Mancoridis, Bec Weeks, Keyon Vafa and Sendhil Mullainathan in 'Potemkin Understanding in Large Language Models', posted to arXiv in June 2025. The name alludes to 'Potemkin villages', sham settlements supposedly erected to impress Empress Catherine II on her 1787 tour of Crimea — a story historians regard as largely legend.
Usage note
Not the same as hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true., which concerns false statements; a potemkin can pair entirely true statements about a concept with incoherent use of it. Related to the stochastic parrotstochastic parrot, n.: A critical label for AI language systems, likening them to parrots that echo patterns in the text they were trained on, fluently but with no grasp of what the words mean. debate, though the authors make a narrower, testable claim.
Related
- jagged intelligencejagged intelligence, n.: The uneven pattern of AI abilities: today's systems can solve very hard problems yet fail at some a child could manage, so their skills form a spiky profile rather than a smooth one.
- stochastic parrotstochastic parrot, n.: A critical label for AI language systems, likening them to parrots that echo patterns in the text they were trained on, fluently but with no grasp of what the words mean.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- fancy autocompletefancy autocomplete, n.: A dismissive description of AI chatbots as nothing more than a souped-up version of the feature on a phone keyboard that suggests your next word.
- Clever Hans effectClever Hans effect, n.: When a person or an AI seems to solve a problem but is really picking up on an unintended clue — named after a horse that seemed to do arithmetic but was reading its questioners' body language.
- hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.
Sources
- Mancoridis, M., Weeks, B., Vafa, K. & Mullainathan, S. (2025). Potemkin Understanding in Large Language Models. arXiv:2506.21521. arxiv.org/abs/2506.21521
Mentioned in
stochastic parrotstochastic parrot, n.: A critical label for AI language systems, likening them to parrots that echo patterns in the text they were trained on, fluently but with no grasp of what the words mean., fancy autocompletefancy autocomplete, n.: A dismissive description of AI chatbots as nothing more than a souped-up version of the feature on a phone keyboard that suggests your next word., jagged intelligencejagged intelligence, n.: The uneven pattern of AI abilities: today's systems can solve very hard problems yet fail at some a child could manage, so their skills form a spiky profile rather than a smooth one., machine understandingmachine understanding, n.: Whether, and in what sense, an AI system grasps what words, situations or problems mean — rather than just producing answers that look as though it does., strawberry problemstrawberry problem, n.: The once-famous failure of AI chatbots to count the r's in 'strawberry' correctly — a simple question that exposed how these models see words in chunks rather than letters.