A3C
In plain wordsA 2016 DeepMind training method in which many copies of an AI agent learn in parallel on separate copies of a task, pooling their updates into one shared neural network.
Asynchronous advantage actor–critic: a deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images. algorithm in which many actor-learner threads, each with its own copy of the environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score., run in parallel on a multi-core CPU and asynchronously apply gradient updates to shared policy and value networks. Each thread collects a few steps of experience, computes n-step advantageadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse. estimates using the valuevalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board. head as critic, and adds an entropy bonusentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early. to discourage premature convergence. Running many actors on separate copies of the environment spreads each round of updates across many different states, breaking up the correlations that experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened. would otherwise have to remove, so A3C can learn on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes. without a replay buffer. Using a multi-core CPU rather than a GPU, it beat earlier methods' overall Atari scores in half the training time, and the same paper applied it to continuous control and to navigating 3D mazes from pixels. A synchronous, batched version (A2C) proved as good and uses GPUs better, and IMPALA later scaled the idea with off-policy corrections.
We ran A3C with 16 worker threads on one machine instead of renting a GPU.
Origin
Introduced by Volodymyr Mnih, Adrià Puigdomènech Badia and colleagues at DeepMind in 'Asynchronous Methods for Deep Reinforcement Learning' (arXiv, February 2016; ICML 2016), which presented asynchronous versions of four algorithms, the advantage actor–critic performing best. OpenAI's Baselines release (2017) popularized the synchronous A2C variant.
Related
- actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic.
- advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse.
- entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early.
- deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images.
- proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
- on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes.
Sources
- Mnih, V. et al. (2016). Asynchronous Methods for Deep Reinforcement Learning. ICML 2016. arXiv:1602.01783. arxiv.org/abs/1602.01783
Mentioned in
actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic., on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes., RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster., entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early., Koray KavukcuogluKoray Kavukcuoglu, prop. n.: A Turkish-born computer scientist who co-authored many of DeepMind's landmark papers and in 2026 took charge of Google DeepMind as its senior vice president., Volodymyr MnihVolodymyr Mnih, prop. n.: A DeepMind researcher who led the 2013–2015 work in which one neural network learned to play dozens of Atari video games from screen pixels, launching deep reinforcement learning.