deep Q-network
In plain wordsA method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level.
A deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images. algorithm and architecture that approximates the optimal action-value functionQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward. with a deep convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. trained by Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next.. The network takes a stack of the last four preprocessed game frames and outputs one Q-value per joystick actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward., so a single forward pass ranks every action. Two devices made training stable: experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened., which samples past transitions at random to break correlations, and a target networktarget network, n.: A slowly updated copy of an AI agent's learning network, used to compute the goals it learns toward, so those goals do not shift every time the network changes., a periodically refreshed copy used to compute the bootstrapped targets. Using one architecture and set of hyperparameters across 49 Atari 2600 games, with rewards clipped to ±1, the 2015 version beat earlier methods on 43 games and scored more than 75 percent of a professional human tester's score on 29. Successors fixed specific flaws — Double DQN (overestimation), prioritized replay, dueling, distributional and noisy networks, combined in Rainbow (2017) — and Agent57 (2020) exceeded the human benchmark on all 57 games.
Our DQN baseline uses the Nature hyperparameters: a one-million-transition replay buffer and target updates every 10,000 steps.
Origin
Introduced by Volodymyr Mnih and colleagues at DeepMind in 'Playing Atari with Deep Reinforcement Learning' (NIPS Deep Learning Workshop, December 2013), which tested seven games; the expanded 'Human-level Control through Deep Reinforcement Learning' (Nature, February 2015) added the target network and 49 games. Its success, soon after AlexNet, launched the deep RL boom.
Related
- Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next.
- experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened.
- target networktarget network, n.: A slowly updated copy of an AI agent's learning network, used to compute the goals it learns toward, so those goals do not shift every time the network changes.
- Arcade Learning EnvironmentArcade Learning Environment, prop. n.: A software platform that lets AI programs play classic Atari 2600 video games, used since 2012 as a standard test of how well one AI can learn many different tasks from screen images.
- deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images.
- deadly triaddeadly triad, n.: The risky combination of three common ingredients in reinforcement learning — approximation, learning from guesses, and learning from other strategies' data — which together can make learning spin out of control.
Sources
- Mnih, V. et al. (2015). Human-level control through deep reinforcement learning. Nature 518, 529–533.
- Mnih, V. et al. (2013). Playing Atari with Deep Reinforcement Learning. NIPS Deep Learning Workshop. arXiv:1312.5602. arxiv.org/abs/1312.5602
- Badia, A. P. et al. (2020). Agent57: Outperforming the Atari Human Benchmark. arXiv:2003.13350. arxiv.org/abs/2003.13350
Mentioned in
Demis HassabisDemis Hassabis, prop. n.: A British AI researcher and former chess prodigy who co-founded DeepMind, led the teams behind AlphaGo and AlphaFold, and shared the 2024 Nobel Prize in Chemistry., reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm., deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images., DeepMindDeepMind, prop. n.: A London artificial intelligence company, bought by Google in 2014, famous for programs that mastered the game of Go and predicted the shapes of proteins. In 2023 it merged into Google DeepMind., epsilon-greedyepsilon-greedy, adj.: A simple rule for balancing trying new things against using what works: most of the time take the best-known option, but a small fraction of the time pick at random., experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened., off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one., Q-functionQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward., Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next., RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster., sample efficiencysample efficiency, n.: How much practice an AI needs to learn something — an agent that learns a game in a hundred tries is far more sample-efficient than one that needs millions., Arcade Learning EnvironmentArcade Learning Environment, prop. n.: A software platform that lets AI programs play classic Atari 2600 video games, used since 2012 as a standard test of how well one AI can learn many different tasks from screen images., Chris WatkinsChris Watkins, prop. n.: A British computer scientist who invented Q-learning in 1989, a simple way for an agent to learn which action is best in each situation purely from trial and error., deadly triaddeadly triad, n.: The risky combination of three common ingredients in reinforcement learning — approximation, learning from guesses, and learning from other strategies' data — which together can make learning spin out of control., deep deterministic policy gradientdeep deterministic policy gradient, n.: A method for teaching AI agents smooth, precise movements, such as controlling a robot arm, by learning a scoring network for actions and steering a decision network toward higher scores., Huber lossHuber loss, n.: A way of measuring prediction errors that treats small errors like squared distance but large ones more gently, so a few wild data points cannot drag the whole model off course., Koray KavukcuogluKoray Kavukcuoglu, prop. n.: A Turkish-born computer scientist who co-authored many of DeepMind's landmark papers and in 2026 took charge of Google DeepMind as its senior vice president., target networktarget network, n.: A slowly updated copy of an AI agent's learning network, used to compute the goals it learns toward, so those goals do not shift every time the network changes., TD-GammonTD-Gammon, prop. n.: A backgammon program built at IBM in the early 1990s that taught itself to play at near world-champion level by playing more than a million games against itself., Volodymyr MnihVolodymyr Mnih, prop. n.: A DeepMind researcher who led the 2013–2015 work in which one neural network learned to play dozens of Atari video games from screen pixels, launching deep reinforcement learning.