Skip to content
Last Dictionary
Plane
Reinforcement LearningTraining & Post-training

rollout

nounAlso: roll-out, playout, roll out

In plain wordsLetting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning.

  1. 1

    Reinforcement learning

    The process, or the resulting trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received., of executing a policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. in an environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score. or simulator from some starting statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next. for a fixed number of steps or until the episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again. ends, in order to collect experience or estimate returnsreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.. In PPOproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.-style training a rollout phase gathers a batch of interaction with the current policy and an update phase learns from it. In large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. RL the rollouts are the sampled responses to training prompts; generating them often dominates the compute budget, so systems separate fast inference 'rollout workers' from the trainer.

    Rollouts took 70 percent of each training step, so we moved generation to a separate inference cluster.

  2. 2

    Planning

    A simulated continuation of play from a position using a fast default policy, repeated many times to estimate the position's value by averaging outcomes — as in the simulation phase of Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average., where it is also called a playout, and in the rollout algorithms of approximate dynamic programming, which improve a base policy by lookahead.

Origin

From backgammon, where 'rolling out' a position means playing it to the end many times with fresh dice to estimate its value. Gerald Tesauro and Gregory Galperin used computer rollouts for on-line policy improvement (NIPS 1996), and Dimitri Bertsekas and colleagues developed 'rollout algorithms' in the late 1990s. Deep RL and language-model training broadened the word to any policy-generated experience.

Usage note

Usage is loose: a rollout can be a single trajectory, a batch of them or the act of generating them. In software deployment the same word means a staged release — an unrelated sense.

Related

See also
  • trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received.
  • episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again.
  • Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average.
  • on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes.
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
  • TD-GammonTD-Gammon, prop. n.: A backgammon program built at IBM in the early 1990s that taught itself to play at near world-champion level by playing more than a million games against itself.

Sources

  1. Tesauro, G. & Galperin, G. R. (1997). On-line Policy Improvement using Monte-Carlo Search. Advances in Neural Information Processing Systems 9 (NIPS 1996).
  2. Bertsekas, D. P., Tsitsiklis, J. N. & Wu, C. (1997). Rollout Algorithms for Combinatorial Optimization. Journal of Heuristics 3, 245–262.

Mentioned in

world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again., Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average., on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples., training looptraining loop, n.: The repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times., trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received., agentic reinforcement learningagentic reinforcement learning, n.: Training an AI assistant by rewarding success on long, multi-step jobs where it must use tools, browse, run code and react to what happens, rather than answer in one go., TD-GammonTD-Gammon, prop. n.: A backgammon program built at IBM in the early 1990s that taught itself to play at near world-champion level by playing more than a million games against itself.