process reward model
In plain wordsA model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
A reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. that scores the intermediate steps of a multi-step solution, typically each line or paragraph of a chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., estimating whether each step is correct or still on a path to a correct answer. A whole solution can be scored by combining step scores, for example as the product of per-step correctness probabilities or as their minimum. PRMs are used to rerank samples in best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers., to guide step-level tree or beam search at inference time, and as dense rewards in reinforcement learning, easing the credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game.. Step labels may come from human annotators, as in the 800,000-label PRM800K dataset, or be estimated automatically by rolling out completions from each step and checking how often they reach the right answer. Drawbacks are the cost and ambiguity of defining a 'step' and vulnerability to reward hacking when used as an RL reward; several 2025 reasoning models were trained with outcome rewards only.
The PRM flagged step 4, where the solution divided by a quantity that could be zero.
Origin
Jonathan Uesato and colleagues at DeepMind (November 2022) compared process- and outcome-based feedback on grade-school math and introduced the abbreviations PRM and ORM. Hunter Lightman and colleagues at OpenAI ('Let's Verify Step by Step', May 2023) showed process supervision beating outcome supervision on MATH, solving 78 percent of a test subset, and released PRM800K; Math-Shepherd (2023) automated step labeling.
Usage note
The training signal is called process supervisionprocess supervision, n.: Training an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts., also valued for safety because it rewards reasoning people can endorse rather than results alone. DeepSeek reported not using PRMs for R1 because of reward hacking and the difficulty of labeling steps.
Related
- outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box.
- process supervisionprocess supervision, n.: Training an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts.
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
- best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game.
- outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box.
Sources
- Uesato, J. et al. (2022). Solving math word problems with process- and outcome-based feedback. arXiv:2211.14275. arxiv.org/abs/2211.14275
- Lightman, H. et al. (2023). Let's Verify Step by Step. arXiv:2305.20050. arxiv.org/abs/2305.20050
Mentioned in
credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game., MATHMATH, prop. n.: A set of 12,500 problems from high-school math competitions, each with a full worked solution, used to test language models on genuinely challenging mathematics., reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples., verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., generative reward modelgenerative reward model, n.: A scoring model that judges an AI's answer by writing out its reasoning in words before giving a verdict, instead of simply outputting a number., outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box., outcome supervisionoutcome supervision, n.: Training an AI by rewarding it only for whether the end result is right — the answer, the passing test, the won game — without grading how it got there., process supervisionprocess supervision, n.: Training an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts., reward shapingreward shaping, n.: Adding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone., rubric rewardrubric reward, n.: A training score built from a checklist of specific things a good answer should do, with an AI grader ticking off each one — a way to reward quality where no single right answer exists.