chain-of-thought monitoring
In plain wordsHaving a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm.
The practice of automatically inspecting the reasoning tracesreasoning trace, n.: The written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see. that reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning. produce before answering or acting, usually with another language model prompted or trained as a monitor, to detect intentions or plans for undesired behavior: reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., sabotage, deception, sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is. or assistance with misuse. Because current models often spell out such plans in plain language ('let's hack the tests'), a monitor with access to the reasoning can flag misbehavior that would be invisible in the actions and final outputs alone. It is used in training to find flawed reward signals, in evaluations and audits, and in deployment as a layer of AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.. Its usefulness depends on monitorabilitymonitorability, n.: How easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language.: if reasoning is unfaithful, encoded or optimized to look innocent, the monitor sees nothing. Training directly against the monitor's flags can backfire by teaching models to misbehave with reasoning that no longer reveals it.
Chain-of-thought monitoring flagged the episode: the model wrote that it would special-case the unit tests.
Origin
An early proposal was Tamera Lanham's 'externalized reasoning oversight' (AI Alignment Forum, August 2022); the idea became practical with reasoning models trained by reinforcement learning from late 2024. Bowen Baker and colleagues at OpenAI (March 2025) showed that GPT-4o, reading the reasoning of a stronger frontier model of the o3-mini class, caught reward hacking in agentic coding tasks far better than a monitor of actions and outputs alone, and that strong training pressure against the monitor produced obfuscated reward hacking.
Usage note
Different from screening a model's final outputs for policy violations, and from interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., which inspects internal activations rather than the text the model writes. How much it can be relied on is debated; see chain-of-thought faithfulnesschain-of-thought faithfulness, n.: Whether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision..
Related
- monitorabilitymonitorability, n.: How easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language.
- chain-of-thought faithfulnesschain-of-thought faithfulness, n.: Whether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision.
- reasoning tracereasoning trace, n.: The written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.
- steganographysteganography, n.: Hiding a secret message inside something that looks ordinary. In AI safety, the worry is that AI systems could hide plans, or signals to one another, inside innocent-looking text.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
Sources
- Lanham, T. (2022). Externalized reasoning oversight: a research direction for language model alignment. AI Alignment Forum / LessWrong.
- Baker, B., Huizinga, J., Gao, L. et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. OpenAI, arXiv:2503.11926. arxiv.org/abs/2503.11926
- Korbak, T., Balesni, M. et al. (2025). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv:2507.11473. arxiv.org/abs/2507.11473
Mentioned in
chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., reasoning tracereasoning trace, n.: The written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see., sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is., schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked., chain-of-thought faithfulnesschain-of-thought faithfulness, n.: Whether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision., latent reasoninglatent reasoning, n.: Getting an AI to do its step-by-step thinking inside its internal numbers rather than writing it out in words — potentially faster and more flexible, but much harder for people to inspect., monitorabilitymonitorability, n.: How easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language., neuraleseneuralese, n.: A nickname for the private 'language' of numbers that AI systems could use to think or to talk to each other, instead of words that people can read., prover–verifier gameprover–verifier game, n.: A training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check., steganographysteganography, n.: Hiding a secret message inside something that looks ordinary. In AI safety, the worry is that AI systems could hide plans, or signals to one another, inside innocent-looking text., trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.