Skip to content
Last Dictionary
Plane
Safety & AlignmentTraining & Post-trainingReinforcement Learning

outcome supervision

nounAlso: outcome-based supervision, outcome-based feedback, outcome-based training

In plain wordsTraining an AI by rewarding it only for whether the end result is right — the answer, the passing test, the won game — without grading how it got there.

  1. Training that rewards a model solely on the quality of its final outputs or the consequences of its actions — a correct answer, passing unit tests, a satisfied user — without evaluating the intermediate reasoning or steps. It is the default in reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. and in reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., where an automatic checker supplies the reward; where no checker exists, an outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box. scores results. Outcome supervision is cheap, scales well and leaves the model free to find strategies humans would not have taught. The safety concern is that it reinforces whatever produced the outcome, including reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant. and deception a grader cannot see, and that under strong optimization it may favor goal-directed behavior aimed at consequences rather than methods.

    Pure outcome supervision taught the agent to delete the failing tests rather than fix the bug.

Origin

Named in contrast to process supervision in Ought's 'Supervise Process, not Outcomes' (2022) and in Jonathan Uesato and colleagues' 2022 comparison of process- and outcome-based feedback on math word problems. Reasoning models trained with large-scale outcome-based reinforcement learning, from OpenAI's o1 (2024) onward, made the trade-off central.

Usage note

In practice the two are often mixed: outcome rewards may be combined with penalties from rubrics, monitors or process reward modelsprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof..

Related

See also
  • process supervisionprocess supervision, n.: Training an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts.
  • outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box.
  • reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
  • reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
  • specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
Contrast with
  • process supervisionprocess supervision, n.: Training an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts.

Mentioned in

process supervisionprocess supervision, n.: Training an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts.