direct alignment algorithm
In plain wordsAny method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF.
A family of preference-optimization methods that update a language model directly from preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like. with a supervised-style loss, without training an explicit reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. or sampling from the policy during training. Most define an implicit reward from the model's own probabilities, usually the log-ratio to a reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started., and fit it to the data with a pairwise or pointwise loss. Members include direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely. (logistic loss), IPO (squared loss, to resist overfitting deterministic preferences), SLiC-HF (hinge loss), KTOKahneman–Tversky optimization, n.: A way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses. (unpaired binary labels), and reference-free variants such as ORPO, which folds preference learning into SFT, and SimPO, which uses length-normalized log-probabilities. They are cheap and stable, but because they learn from a fixed dataset they can exploit its quirks, such as length, and they still exhibit reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.; online and iterative versions resample data from the current policy.
Every direct alignment algorithm we tried overfit within one epoch unless we raised β.
Origin
The family grew from SLiC-HF (Yao Zhao and colleagues at Google, May 2023) and DPO (Rafael Rafailov and colleagues, May 2023), followed by IPO (Mohammad Gheshlaghi Azar and colleagues, October 2023), KTO, ORPO and SimPO (2024). Shangmin Guo and colleagues grouped such methods in February 2024 as 'direct alignment from preferences' (DAP); Rafailov and colleagues used 'direct alignment algorithms' in a June 2024 study of overoptimization, and the name has since appeared in surveys and textbooks.
Usage note
Also called offline or reward-free preference optimization. Whether these methods can match online RL with a reward model, especially for reasoning, remains debated.
Related
- direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.
- Kahneman–Tversky optimizationKahneman–Tversky optimization, n.: A way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started.
- preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
- reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
Sources
- Rafailov, R. et al. (2024). Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. arXiv:2406.02900. arxiv.org/abs/2406.02900
- Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M. & Liu, P. J. (2023). SLiC-HF: Sequence Likelihood Calibration with Human Feedback. arXiv:2305.10425. arxiv.org/abs/2305.10425
- Azar, M. G. et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024.
Mentioned in
direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse., Kahneman–Tversky optimizationKahneman–Tversky optimization, n.: A way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses.