intrinsic motivation
In plain wordsGiving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet.
In reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., the use of reward signals generated by the agent itself rather than by the task — for novelty, surprise, learning progress, competence or control over the environment — to drive exploration and skill acquisition, especially when external rewards are sparsesparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success. or absent. Common forms include count-based bonuses for visiting rarely seen states (extended to large spaces with pseudo-counts), prediction-error curiosity that pays where a learned dynamics model is wrong (the Intrinsic Curiosity Module; Random Network Distillation), information gain about a world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., and empowerment. Intrinsic rewards are usually added to the extrinsic reward with a weight. A known failure is the 'noisy TV' problem: an agent paid for unpredictability can be captivated by pure randomness, so better methods reward learnable novelty. Random Network Distillation (2018) was reported by its authors as the first method to exceed average human performance on the hard-exploration Atari game Montezuma's Revenge without demonstrations or access to the game's internal state.
With a curiosity bonus the agent explored the whole maze before it ever found the exit reward.
Origin
The psychological concept — activity pursued for its own sake — was developed by Robert White (1959) and by Edward Deci and Richard Ryan from the 1970s. Jürgen Schmidhuber built curiosity rewards into learning agents in 1991, and Satinder Singh, Andrew Barto and Nuttapong Chentanez's 'Intrinsically Motivated Reinforcement Learning' (NIPS 2004) named the RL research program.
Usage note
Psychology contrasts intrinsic with extrinsic motivation (acting for external rewards). RL borrows the distinction loosely: an 'intrinsic' reward is still a formula chosen by a designer.
Related
- exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one.
- sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success.
- reward shapingreward shaping, n.: Adding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone.
- world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.
- hierarchical reinforcement learninghierarchical reinforcement learning, n.: Reinforcement learning organized in layers, where a high-level part chooses goals or skills — 'go to the kitchen' — and lower-level parts handle the detailed actions to achieve them.
- Arcade Learning EnvironmentArcade Learning Environment, prop. n.: A software platform that lets AI programs play classic Atari 2600 video games, used since 2012 as a standard test of how well one AI can learn many different tasks from screen images.
Sources
- Singh, S., Barto, A. G. & Chentanez, N. (2004). Intrinsically Motivated Reinforcement Learning. Advances in Neural Information Processing Systems 17.
- Pathak, D., Agrawal, P., Efros, A. A. & Darrell, T. (2017). Curiosity-driven Exploration by Self-supervised Prediction. ICML 2017.
- Burda, Y., Edwards, H., Storkey, A. & Klimov, O. (2019). Exploration by Random Network Distillation. ICLR 2019. arXiv:1810.12894.
Mentioned in
exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one., Jürgen SchmidhuberJürgen Schmidhuber, prop. n.: A German computer scientist who co-invented the LSTM, a neural network with memory that powered much of speech recognition and translation in the 2010s, and who argues that early pioneers go under-credited., sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success., Andrew BartoAndrew Barto, prop. n.: An American computer scientist who, with his student Richard Sutton, laid the foundations of reinforcement learning, the study of how machines learn by trial and error from rewards., Arcade Learning EnvironmentArcade Learning Environment, prop. n.: A software platform that lets AI programs play classic Atari 2600 video games, used since 2012 as a standard test of how well one AI can learn many different tasks from screen images., hierarchical reinforcement learninghierarchical reinforcement learning, n.: Reinforcement learning organized in layers, where a high-level part chooses goals or skills — 'go to the kitchen' — and lower-level parts handle the detailed actions to achieve them., open-endednessopen-endedness, n.: The ability of a system to keep producing new and interesting things without end — as evolution and human culture do — instead of stopping once it solves one fixed task., reward shapingreward shaping, n.: Adding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone., upper confidence boundupper confidence bound, n.: A rule for choosing among uncertain options that gives each one a bonus for how little is known about it, so promising but rarely tried options get a fair chance.