Skip to content
Last Dictionary
Plane
Reinforcement Learning

Dyna

proper noun/DYE-nuh/Also: Dyna architecture, Dyna-Q

In plain wordsAn early design in which a learning agent builds a model of its world from real experience, then also practices on imagined experience from that model, squeezing more learning out of each real step.

  1. An agent architecture, proposed by Richard Sutton, that folds planning into reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.: real experience feeds direct updates (for example Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next.) and also trains a model of the environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score., and between real steps the agent makes extra updates on transitions simulated by that model. In Dyna-Q, each real step is followed by n planning updates on previously seen state–action pairs replayed through the model, which sharply reduces the number of real interactions needed in small problems. Prioritized sweeping focuses those updates where values are changing most. Dyna showed that planning and learning can share a single value-update mechanism, and it is the template for modern model-basedmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting. methods that train a policy on rollouts of a learned world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real..

    With 50 planning steps per real step, the Dyna-Q agent found the maze's shortest path within a few episodes.

Origin

Introduced by Richard Sutton in 'Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming' (ICML 1990) and summarized in 'Dyna, an Integrated Architecture for Learning, Planning, and Reacting' (SIGART Bulletin, 1991). Andrew Moore and Christopher Atkeson, and independently Jing Peng and Ronald Williams, added prioritized sweeping in 1993.

Related

See also
  • model-based reinforcement learningmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting.
  • world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.
  • Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next.
  • experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened.
  • dynamic programmingdynamic programming, n.: A problem-solving method that breaks a big problem into smaller overlapping pieces, solves each piece once and reuses the answers — the basis for computing the best long-term strategy step by step.
Contrast with
  • model-free reinforcement learningmodel-free reinforcement learning, n.: Learning to act well purely from trial and error, without ever building an internal picture of how the world works — like learning a game by habit rather than by thinking moves ahead.

Sources

  1. Sutton, R. S. (1991). Dyna, an Integrated Architecture for Learning, Planning, and Reacting. ACM SIGART Bulletin 2(4), 160–163.

Mentioned in

world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., model-based reinforcement learningmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting.