self-taught reasoner
In plain wordsA training method in which a language model writes step-by-step explanations for problems, keeps the ones that led to correct answers, and learns from them, so it gradually teaches itself to reason.
A method by which a language model, given only a handful of worked examples, bootstraps its own skill at writing useful rationales (chains of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.). It repeats a simple cycle: attempt every training question with a written rationale and keep those whose final answers are right; for the questions missed, reveal the answer and ask the model to explain it ('rationalization'), keeping those explanations as well; fine-tune on everything kept; and begin again. STaR can be viewed as an approximation to a policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. with a binary correctness reward, and as a form of expert iterationexpert iteration, n.: A learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter. or rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work. fine-tuning. Its authors reported that on CommonsenseQA a STaR-trained 6-billion-parameter GPT-J performed comparably to a fine-tuned model about 30 times larger. Successors include Quiet-STaR, which learns rationales for arbitrary text, and V-STaR, which also trains a verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded..
After three STaR iterations the rationales were shorter and arithmetic accuracy had climbed eleven points.
Origin
Introduced by Eric Zelikman, Yuhuai Wu, Jesse Mu and Noah Goodman (Stanford and Google) in March 2022 and published at NeurIPS 2022. It is widely cited as a forerunner of reinforcement-learning-trained reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning..
Usage note
Rationalization can yield post-hoc justifications that fit a known answer without showing how it could be derived, one reason later methods prefer purely outcome-filtered sampling or RL.
Related
- expert iterationexpert iteration, n.: A learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter.
- rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
Sources
- Zelikman, E., Wu, Y., Mu, J. & Goodman, N. D. (2022). STaR: Bootstrapping Reasoning With Reasoning. NeurIPS 2022. arXiv:2203.14465. arxiv.org/abs/2203.14465
Mentioned in
rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work., self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples., expert iterationexpert iteration, n.: A learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter.