episode
In plain wordsOne complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again.
A finite sequence of agent–environment interactions that begins in an initial statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next. and ends in a terminal state or at a time limit, after which the environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score. is reset. Tasks with natural endings — games, puzzles, a robot's grasp attempt, a single model response — are called episodic, as opposed to continuing tasks, such as process control, that run indefinitely. Episodes are the unit over which returnsreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible. are summed, Monte Carlo estimates are formed and learning curves are usually reported. Modern interfaces such as GymnasiumGymnasium, prop. n.: A free software library that gives reinforcement learning programs one standard way to plug into practice tasks, from simple games to simulated robots, so methods can be tried on identical problems. It succeeded OpenAI's Gym. distinguish termination (the task's own end, after which future value is zero) from truncation (an artificial time limit, after which value should still be estimated); conflating the two biases value estimates. In large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. RL an episode is typically one prompt and the full response or multi-turn interaction it produces.
Each episode ends after 1,000 steps or when the robot falls over.
Origin
Psychology's 'trial' was the older word for one run of a repeated learning task, and early RL papers used it too. 'Episode' became the usual RL word during the 1990s, and the episodic/continuing distinction was formalized in Sutton and Barto's textbook (1998).
Usage note
Not to be confused with episodic memory in cognitive science, or with the 'episodes' of meta-learningmeta-learning, n.: Training a system across many different tasks so that it gets better at learning itself, letting it pick up a new task quickly from only a few examples., where an episode is a small sampled training task.
Related
- trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received.
- rolloutrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning.
- returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.
- environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score.
- GymnasiumGymnasium, prop. n.: A free software library that gives reinforcement learning programs one standard way to plug into practice tasks, from simple games to simulated robots, so methods can be tried on identical problems. It succeeded OpenAI's Gym.
- sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success.
- trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received.
Mentioned in
bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data., environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score., GymnasiumGymnasium, prop. n.: A free software library that gives reinforcement learning programs one standard way to plug into practice tasks, from simple games to simulated robots, so methods can be tried on identical problems. It succeeded OpenAI's Gym., returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible., rolloutrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning., sample efficiencysample efficiency, n.: How much practice an AI needs to learn something — an agent that learns a game in a hundred tries is far more sample-efficient than one that needs millions., temporal difference learningtemporal difference learning, n.: A way of learning predictions by adjusting each guess toward the next, slightly better-informed guess instead of waiting for the final outcome — like revising a weather forecast every day as the date approaches., trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received., REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly.