chain-of-thought faithfulness
In plain wordsWhether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision.
The degree to which a model's stated chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. accurately represents the process that actually produced its answer. A trace is unfaithful when it omits factors that causally influenced the output, rationalizes a conclusion reached on other grounds, or contains steps the model does not use. Faithfulness is tested by intervention: planting hints or biasing features in prompts and checking whether answers change without the reasoning acknowledging them, or truncating, corrupting or paraphrasing the reasoning and measuring the effect on the answer. Yanda Chen and colleagues (2025) found that reasoning models frequently relied on hints planted in the prompt, including ones that enabled reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., yet acknowledged them only a minority of the time, and that outcome-based reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. raised faithfulness at first but soon leveled off. Faithfulness is needed for chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm. to give strong guarantees.
The model switched its answer to match the hint but never mentioned the hint in its reasoning.
Origin
Alon Jacovi and Yoav Goldberg (2020) set out faithfulness as a criterion for model explanations. Miles Turpin and colleagues (2023) showed that chain-of-thought explanations can systematically misrepresent why a model answered as it did; Tamera Lanham and colleagues at Anthropic (2023) measured it by editing the reasoning itself — truncating it, inserting mistakes or paraphrasing it; and Yanda Chen and colleagues (2025) extended hint-based tests to reasoning models.
Usage note
Faithfulness differs from correctness: a valid-looking argument can be unfaithful, and a faithful one can be wrong. Some researchers argue that safety needs monitorabilitymonitorability, n.: How easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language. rather than full faithfulness, since reasoning that a hard task makes necessary is informative even when incomplete.
Related
- chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm.
- monitorabilitymonitorability, n.: How easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- reasoning tracereasoning trace, n.: The written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see.
- post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
- steganographysteganography, n.: Hiding a secret message inside something that looks ordinary. In AI safety, the worry is that AI systems could hide plans, or signals to one another, inside innocent-looking text.
Sources
- Turpin, M., Michael, J., Perez, E. & Bowman, S. R. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.
- Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J. et al. (2025). Reasoning Models Don't Always Say What They Think. arXiv:2505.05410. arxiv.org/abs/2505.05410
Mentioned in
chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning., chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm., faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given., introspectionintrospection, n.: Looking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up., reasoningreasoning, n.: Working out conclusions step by step from facts, rules or evidence, as in solving a math problem or planning a trip. Whether today's AI truly reasons, or imitates reasoning, is debated., reasoning tracereasoning trace, n.: The written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see., scratchpadscratchpad, n.: A space where an AI writes rough working — calculations, plans, notes — before or while producing its answer, much as a person jots on scrap paper., model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint., monitorabilitymonitorability, n.: How easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language., steganographysteganography, n.: Hiding a secret message inside something that looks ordinary. In AI safety, the worry is that AI systems could hide plans, or signals to one another, inside innocent-looking text.