alignment tax
In plain wordsThe price of caution in AI: whatever performance, money or time a developer sacrifices to make a system safe and aligned, compared with an otherwise identical system built with no such care.
The difference in capability, compute, development time, latency or money between an AI system built to be aligned and an otherwise equivalent system built without that effort. The concept matters strategically: if the tax is high, competitive pressure pushes developers to skip alignment measures, so lowering it is itself a safety goal. In practice the term often refers to measured capability regressions from alignment training — the InstructGPT paper (2022) reported that RLHF degraded performance on some public NLP benchmarks and reduced the regression by mixing pretraining updates into fine-tuning. A negative alignment tax, in which alignment work also makes a system more useful (as RLHF did for following instructions), is often cited to explain why laboratories adopt alignment techniques readily when they also improve products.
If the monitoring layer adds 40% to inference cost, that's an alignment tax many startups won't pay.
Origin
Popularized by Paul Christiano, who used it from about 2019 for the competitive cost of alignment; it appears in Amanda Askell and colleagues' 2021 study of language assistants and in OpenAI's InstructGPT paper by Long Ouyang and colleagues (2022).
Usage note
Whether current alignment work imposes a net tax depends on what is measured — benchmark scores, usefulness to customers or reduction of catastrophic risk.
Related
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- AI arms raceAI arms race, n.: A competition among countries or companies to build the most powerful AI first, in which fear of falling behind pushes everyone to move faster and cut corners on safety.
- post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner.
- catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on.
Sources
- Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. arXiv:2203.02155. arxiv.org/abs/2203.02155
- Askell, A., Bai, Y., Chen, A. et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861. arxiv.org/abs/2112.00861
Mentioned in
catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on., InstructGPTInstructGPT, prop. n.: A 2022 OpenAI language model trained with human feedback to follow instructions; people preferred its answers to those of a model 100 times larger, and its recipe led directly to ChatGPT., over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program., safetywashingsafetywashing, n.: Making AI work sound as if it improves safety when it mostly just makes AI more capable — like 'greenwashing' for environmental claims.