exploration–exploitation trade-off
In plain wordsThe choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one.
The tension faced by any learner that must act in order to gather information: every choice of the actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward. that looks best on current evidence (exploitation) forgoes a chance to learn about alternatives that might prove better, and every trial of an uncertain alternative (exploration) costs reward that the apparent favorite would probably have earned. Too little exploration locks an agent into a suboptimal habit; too much wastes reward on options already known to be poor. The dilemma is formalized most cleanly in the multi-armed banditmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others., where strategies are compared by regretregret, n.: How much worse a learner did than it could have done had it known the best choice from the start — the price of having to learn by trying. and principled solutions include optimism (upper confidence boundupper confidence bound, n.: A rule for choosing among uncertain options that gives each one a bonus for how little is known about it, so promising but rarely tried options get a fair chance. methods), Thompson samplingThompson sampling, n.: A way to choose among uncertain options by picking each one about as often as it is likely to be the best, based on everything learned so far. and, for discounted Bayesian bandits, the Gittins index. In full reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., simple schemes such as ε-greedyepsilon-greedy, adj.: A simple rule for balancing trying new things against using what works: most of the time take the best-known option, but a small fraction of the time pick at random. and Boltzmann (softmax) action selection or entropy bonusesentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early. are common, while hard-exploration problems with sparse rewardssparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success. call for count-based bonuses or curiosityintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet..
We raised the entropy bonus because the agent stopped exploring after finding the first small reward.
Origin
The dilemma appears in sequential experimental design — William Thompson's 1933 paper and Herbert Robbins's 1952 bandit formulation — and in adaptive control as Alexander Fel'dbaum's 'dual control' (c. 1960). The phrase 'exploration versus exploitation' was popularized by John Holland's Adaptation in Natural and Artificial Systems (1975) and, in management science, by James March (1991), before becoming standard in RL.
Usage note
In large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. RL, a debated 2025 finding by Yang Yue and colleagues is that RL mostly sharpens answers the base model could already sample rather than discovering new ones, making exploration a central open problem for reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds..
Related
- multi-armed banditmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others.
- epsilon-greedyepsilon-greedy, adj.: A simple rule for balancing trying new things against using what works: most of the time take the best-known option, but a small fraction of the time pick at random.
- upper confidence boundupper confidence bound, n.: A rule for choosing among uncertain options that gives each one a bonus for how little is known about it, so promising but rarely tried options get a fair chance.
- Thompson samplingThompson sampling, n.: A way to choose among uncertain options by picking each one about as often as it is likely to be the best, based on everything learned so far.
- intrinsic motivationintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet.
- regretregret, n.: How much worse a learner did than it could have done had it known the best choice from the start — the price of having to learn by trying.
- entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early.
Sources
- Holland, J. H. (1975). Adaptation in Natural and Artificial Systems. University of Michigan Press.
- March, J. G. (1991). Exploration and Exploitation in Organizational Learning. Organization Science 2(1), 71–87.
- Yue, Y. et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837.
Mentioned in
reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., epsilon-greedyepsilon-greedy, adj.: A simple rule for balancing trying new things against using what works: most of the time take the best-known option, but a small fraction of the time pick at random., multi-armed banditmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others., sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success., entropy collapseentropy collapse, n.: A failure in reinforcement-learning training where an AI quickly becomes overconfident and stops trying different answers, so it explores less and its improvement stalls., entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early., intrinsic motivationintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet., John HollandJohn Holland, prop. n.: An American scientist who invented genetic algorithms, which solve problems by breeding and mutating populations of candidate answers, and helped found the science of complex systems., regretregret, n.: How much worse a learner did than it could have done had it known the best choice from the start — the price of having to learn by trying., Thompson samplingThompson sampling, n.: A way to choose among uncertain options by picking each one about as often as it is likely to be the best, based on everything learned so far., upper confidence boundupper confidence bound, n.: A rule for choosing among uncertain options that gives each one a bonus for how little is known about it, so promising but rarely tried options get a fair chance.