metacognition
In plain wordsThinking about one's own thinking: knowing what you know and don't know, judging how confident to be, and noticing when you have made a mistake.
Cognition about one's own cognitive processes, comprising monitoring — assessing one's own knowledge, confidence, memory or errors, as in feeling-of-knowing and confidence judgments — and control, using those assessments to guide behavior by seeking information, checking work, abstaining or changing strategy. Psychologists measure it by how well confidence tracks accuracy (metacognitive sensitivity). In AI, analogous capacities include a model's calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., its ability to predict whether it knows an answer, to abstain or ask for clarification, to detect errors in its own reasoning, and to report its internal states (introspectionintrospection, n.: Looking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up.); reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning. that check and revise their chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. show a functional form of metacognitive control. Metacognition is central to higher-order theories of consciousnesshigher-order theories of consciousness, n.: Theories holding that a mental state is conscious when the mind, in effect, notices it — when some further thought or perception represents you as being in that state., but its presence in a system is not by itself evidence of experience.
Origin
Introduced by the developmental psychologist John H. Flavell in the mid-1970s (in print by 1976, in a chapter on metacognitive aspects of problem solving), extending his earlier term 'metamemory'; his 'Metacognition and Cognitive Monitoring' (American Psychologist, 1979) is the standard reference. Thomas Nelson and Louis Narens's monitoring-and-control framework (1990) shaped later research.
Related
- introspectionintrospection, n.: Looking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up.
- calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- higher-order theories of consciousnesshigher-order theories of consciousness, n.: Theories holding that a mental state is conscious when the mind, in effect, notices it — when some further thought or perception represents you as being in that state.
- theory of mindtheory of mind, n.: Being able to work out what someone else believes, wants or knows — even when it is not what you believe — and to use that to guess what they will do next. Researchers test whether AI has it.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
- System 1 and System 2System 1 and System 2, phr.: A popular way of describing two modes of thinking: fast, automatic and intuitive (System 1) versus slow, effortful and deliberate (System 2). AI researchers borrow it to contrast instant answers with step-by-step reasoning.
Sources
- Flavell, J. H. (1979). Metacognition and Cognitive Monitoring: A New Area of Cognitive–Developmental Inquiry. American Psychologist 34(10), 906–911.
- Kadavath, S. et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221.
Mentioned in
introspectionintrospection, n.: Looking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up., self-awarenessself-awareness, n.: Awareness of oneself as oneself — knowing that you exist, what you are and what state you are in. People often ask whether AI is self-aware, but the phrase can mean several quite different things., System 1 and System 2System 1 and System 2, phr.: A popular way of describing two modes of thinking: fast, automatic and intuitive (System 1) versus slow, effortful and deliberate (System 2). AI researchers borrow it to contrast instant answers with step-by-step reasoning., higher-order theories of consciousnesshigher-order theories of consciousness, n.: Theories holding that a mental state is conscious when the mind, in effect, notices it — when some further thought or perception represents you as being in that state.