machine understanding
In plain wordsWhether, and in what sense, an AI system grasps what words, situations or problems mean — rather than just producing answers that look as though it does.
The capacity of an AI system to grasp meaning — of language, situations, problems or its own outputs — in a sense that goes beyond producing appropriate responses; also the long-running debate over whether machines, and now large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., have it. Functional views treat understanding as robust, flexible and generalizable competence that can be tested. Representational views look for internal world modelsworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real. and structured concepts, as in probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere. and mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior. studies. Other views require groundinggrounding, n.: Connecting an AI's words to something real — trusted documents, a live search, images or the physical world — so that its answers rest on evidence rather than only on what it absorbed in training. in perception and action, original intentionalityintentionality, n.: The way thoughts, beliefs and desires are about things: a belief is about rain, a wish is for cake. Philosophers ask whether a machine's internal states can be about anything. or consciousness, and conclude that text-only models cannot understand, as argued through the Chinese roomChinese room, n.: A thought experiment: a person who knows no Chinese follows a rulebook to answer Chinese messages so well that outsiders think they understand. If the person doesn't understand, does a computer running a program?, the symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words. and the octopus testoctopus test, n.: A thought experiment in which a clever octopus learns to impersonate a stranded islander by eavesdropping on an undersea cable, without ever knowing what the words refer to. It asks whether text-trained AI can grasp meaning.. Many researchers now treat understanding as graded and multidimensional, and cite collapses under small rewordings as evidence that some apparent understanding is shallow.
A model that aces the reading test but fails a reworded version tells us something about the limits of machine understanding.
Origin
The question goes back to Alan Turing (1950) and John Searle's Chinese roomChinese room, n.: A thought experiment: a person who knows no Chinese follows a rulebook to answer Chinese messages so well that outsiders think they understand. If the person doesn't understand, does a computer running a program? (1980), and 'natural language understanding' was a named goal of AI research by the early 1970s. The debate over whether language models understand intensified after Emily Bender and Alexander Koller's octopus testoctopus test, n.: A thought experiment in which a clever octopus learns to impersonate a stranded islander by eavesdropping on an undersea cable, without ever knowing what the words refer to. It asks whether text-trained AI can grasp meaning. (2020) and the public release of ChatGPTChatGPT, prop. n.: OpenAI's AI chatbot, released free to the public in November 2022, which answers questions, writes and converses in everyday language. Its sudden popularity began the current AI boom. in 2022.
Usage note
A 2022 survey of natural-language-processing researchers found them split almost evenly on whether a model trained only on text could understand language in some non-trivial sense. Potemkin understandingPotemkin understanding, n.: When an AI can correctly explain a concept yet cannot use it properly, revealing that its apparent understanding is a facade — like the fake village fronts said to have been built to impress a Russian empress. and stochastic parrotstochastic parrot, n.: A critical label for AI language systems, likening them to parrots that echo patterns in the text they were trained on, fluently but with no grasp of what the words mean. are critical terms from the same debate.
Related
- Chinese roomChinese room, n.: A thought experiment: a person who knows no Chinese follows a rulebook to answer Chinese messages so well that outsiders think they understand. If the person doesn't understand, does a computer running a program?
- octopus testoctopus test, n.: A thought experiment in which a clever octopus learns to impersonate a stranded islander by eavesdropping on an undersea cable, without ever knowing what the words refer to. It asks whether text-trained AI can grasp meaning.
- symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words.
- natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text.
- world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.
- stochastic parrotstochastic parrot, n.: A critical label for AI language systems, likening them to parrots that echo patterns in the text they were trained on, fluently but with no grasp of what the words mean.
- Potemkin understandingPotemkin understanding, n.: When an AI can correctly explain a concept yet cannot use it properly, revealing that its apparent understanding is a facade — like the fake village fronts said to have been built to impress a Russian empress.
- natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text.
Sources
- Mitchell, M. & Krakauer, D. C. (2023). The Debate over Understanding in AI's Large Language Models. Proceedings of the National Academy of Sciences 120(13).
- Michael, J., Holtzman, A., Parrish, A. et al. (2022). What Do NLP Researchers Believe? Results of the NLP Community Metasurvey. arXiv:2208.12852. arxiv.org/abs/2208.12852
Mentioned in
large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., Chinese roomChinese room, n.: A thought experiment: a person who knows no Chinese follows a rulebook to answer Chinese messages so well that outsiders think they understand. If the person doesn't understand, does a computer running a program?, Douglas HofstadterDouglas Hofstadter, prop. n.: An American cognitive scientist whose prize-winning book 'Gödel, Escher, Bach' inspired a generation to study minds and AI, and who studies analogy as the heart of thinking., Emily M. BenderEmily M. Bender, prop. n.: An American linguist who argues that language models only mimic the form of language without understanding it, and who co-coined the phrase 'stochastic parrots'., John SearleJohn Searle, prop. n.: An American philosopher who devised the Chinese room argument, which holds that a computer running a program can manipulate symbols without understanding anything., natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text., Noam ChomskyNoam Chomsky, prop. n.: An American linguist whose theories of grammar reshaped linguistics and cognitive science, and who argues that chatbots like ChatGPT imitate language without understanding it., philosophy of artificial intelligencephilosophy of artificial intelligence, n.: The branch of philosophy that asks what AI is and could be: whether machines can think, understand or be conscious, what intelligence is, and what we owe to and should expect from intelligent machines., strong AIstrong AI, n.: Originally, the claim that a computer running the right program would literally have a mind and understand, not just imitate understanding. Today it often simply means AI as capable as humans across the board., symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words., theory of mindtheory of mind, n.: Being able to work out what someone else believes, wants or knows — even when it is not what you believe — and to use that to guess what they will do next. Researchers test whether AI has it., BlockheadBlockhead, n.: An imagined machine that passes any conversation test by looking up prewritten replies in an unimaginably huge table. Most people judge it unintelligent, which suggests that behavior alone does not prove intelligence., intentionalityintentionality, n.: The way thoughts, beliefs and desires are about things: a belief is about rain, a wish is for cake. Philosophers ask whether a machine's internal states can be about anything., octopus testoctopus test, n.: A thought experiment in which a clever octopus learns to impersonate a stranded islander by eavesdropping on an undersea cable, without ever knowing what the words refer to. It asks whether text-trained AI can grasp meaning.