action
In plain wordsA choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward.
- 1
Reinforcement learning
The output an agentagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks. emits at each time step, written a or A_t, chosen from an action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm. according to its policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards.. The action, together with the current statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next., determines the probability distribution over the next state and the rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time.. Actions may be discrete (one of 18 joystick commands in Atari games), continuous (a torque for each robot joint) or structured (a whole sequence of tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. when a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. is trained as a policy). Their timescale is a design choice: hierarchical reinforcement learninghierarchical reinforcement learning, n.: Reinforcement learning organized in layers, where a high-level part chooses goals or skills — 'go to the kitchen' — and lower-level parts handle the detailed actions to achieve them. lets an agent choose temporally extended actions, called options, as well as primitive ones.
At each step the policy outputs an action vector of seven joint velocities.
- 2
Agents
An operation an AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. performs on the outside world — a tool calltool call, n.: A single request from an AI model to use one of its tools — naming the tool and filling in the details — which the surrounding software then carries out and reports back on., a shell command, a click in a computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people. session, a sent message — as opposed to text it merely generates. Agent-safety practice distinguishes reversible from irreversible actions and often requires human approval for the latter.
Origin
Inherited from decision theory and Richard Bellman's dynamic programming (1950s), in which a decision or action is selected at each stage of a process. RL's state–action–reward vocabulary was fixed by Sutton and Barto's textbook (1998).
Related
- action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm.
- policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards.
- statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next.
- rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time.
- tool calltool call, n.: A single request from an AI model to use one of its tools — naming the tool and filling in the details — which the surrounding software then carries out and reports back on.
- action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer.
- statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next.
Mentioned in
world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm., actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic., advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse., agentagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks., credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game., deep Q-networkdeep Q-network, n.: A method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level., environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score., epsilon-greedyepsilon-greedy, adj.: A simple rule for balancing trying new things against using what works: most of the time take the best-known option, but a small fraction of the time pick at random., exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one., Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice., Markov propertyMarkov property, n.: The assumption that the future depends only on the present situation, not on how you got there — so knowing the current state is enough to predict what happens next., model-based reinforcement learningmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting., policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards., Q-functionQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward., Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next., reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do., statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next., trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received., Arcade Learning EnvironmentArcade Learning Environment, prop. n.: A software platform that lets AI programs play classic Atari 2600 video games, used since 2012 as a standard test of how well one AI can learn many different tasks from screen images., decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action., Thompson samplingThompson sampling, n.: A way to choose among uncertain options by picking each one about as often as it is likely to be the best, based on everything learned so far.