state
In plain wordsA snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next.
- 1
Reinforcement learning
A description of the environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score. at a time step, written s or S_t, that together with the chosen actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward. determines the probabilities of what happens next. A state with this Markov propertyMarkov property, n.: The assumption that the future depends only on the present situation, not on how you got there — so knowing the current state is enough to predict what happens next. summarizes all relevant history, which is what makes Markov decision processesMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice. tractable. States may be discrete (board positions), continuous (a robot's joint angles and velocities) or high-dimensional (raw pixels); the set of all of them is the state space. When the agent receives only partial or noisy observations, it must act on a belief state or a memory of past observations instead, as in a partially observable Markov decision processpartially observable Markov decision process, n.: A version of the standard decision-making model in which the decision-maker cannot see the whole situation — like a poker player who cannot see other hands — and must act on clues and memory.. In deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images. a network's internal representation often serves as a learned state.
Stacking the last four frames gives the agent a state that captures the ball's velocity.
- 2
Neural networks
Short for hidden state: the vector that a recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. or state space modelstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer. carries from one step of a sequence to the next, summarizing everything it has processed so far.
Origin
Borrowed from physics, automata theory and control engineering, where Rudolf Kálmán's state-space methods (c. 1960) describe a system by a vector of state variables. Richard Bellman's dynamic programming (1950s) made the state the basic unit of sequential decision problems, and RL inherited his usage.
Related
- Markov propertyMarkov property, n.: The assumption that the future depends only on the present situation, not on how you got there — so knowing the current state is enough to predict what happens next.
- Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice.
- partially observable Markov decision processpartially observable Markov decision process, n.: A version of the standard decision-making model in which the decision-maker cannot see the whole situation — like a poker player who cannot see other hands — and must act on clues and memory.
- action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm.
- environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score.
- value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board.
- actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward.
Sources
- Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.), ch. 3. MIT Press.
Mentioned in
world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward., action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm., advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse., agentagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks., behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., Bellman equationBellman equation, n.: A rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time., environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score., episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again., Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice., Markov propertyMarkov property, n.: The assumption that the future depends only on the present situation, not on how you got there — so knowing the current state is enough to predict what happens next., model-based reinforcement learningmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting., Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average., multi-armed banditmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others., perceptionperception, n.: How an AI system or robot turns raw sensor signals, such as camera pixels, microphone sound or laser scans, into an understanding of what is around it: objects, people, words and distances., policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards., Q-functionQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward., Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next., reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do., rolloutrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning., temporal difference learningtemporal difference learning, n.: A way of learning predictions by adjusting each guess toward the next, slightly better-informed guess instead of waiting for the final outcome — like revising a weather forecast every day as the date approaches., trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received., value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board., decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action.