AlphaStar
In plain wordsA DeepMind AI that learned to play the real-time strategy game StarCraft II well enough to reach its highest public rank, Grandmaster, in 2019.
A deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images. system for the real-time strategy game StarCraft II, a long-standing AI challenge because of its combinatorially huge action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm., simultaneous real-time play, games lasting thousands of decisions and imperfect information (the 'fog of war'). Its policy, a deep network combining a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. over game units, a recurrent LSTMlong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences. core and pointer-network action heads, was first trained by imitating human replays and then improved by multi-agent reinforcement learningmulti-agent reinforcement learning, n.: Reinforcement learning with several AI agents learning at once in a shared world, where each must adapt to the others — as teammates, rivals or both. in a 'league': a growing population of agents, including dedicated exploiters trained to find weaknesses in the main agents, with matchmaking that preserved the diversity that naive self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples. loses. Restricted to a camera view and action rates comparable to human players, the final agent was rated Grandmaster, the top league, with each of the game's three races on the public Battle.net ladder, placing it above 99.8 percent of ranked players.
Like AlphaStar's league, our training keeps exploiter agents whose only job is to beat the current champion.
Origin
Developed at DeepMind by a team led by Oriol Vinyals. An early version beat the professional players Dario 'TLO' Wünsch and Grzegorz 'MaNa' Komincz in matches revealed in January 2019; the final system was described in 'Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning' (Nature, October 2019).
Usage note
The January 2019 games drew criticism that superhuman speed and click precision, rather than strategy, explained the wins, which led DeepMind to impose stricter human-like limits on the final agent.
Related
- multi-agent reinforcement learningmulti-agent reinforcement learning, n.: Reinforcement learning with several AI agents learning at once in a shared world, where each must adapt to the others — as teammates, rivals or both.
- self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.
- deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images.
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- OpenAI FiveOpenAI Five, prop. n.: A team of five AI players built by OpenAI that learned the team video game Dota 2 by playing against itself, and in April 2019 beat the reigning world-champion human team.
- AlphaZeroAlphaZero, prop. n.: A 2017 DeepMind program that taught itself chess, shogi and Go from nothing but the rules by playing against itself, and within hours of training beat the strongest existing programs.
Sources
- Vinyals, O. et al. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354.
Mentioned in
DeepMindDeepMind, prop. n.: A London artificial intelligence company, bought by Google in 2014, famous for programs that mastered the game of Go and predicted the shapes of proteins. In 2023 it merged into Google DeepMind., OpenAI FiveOpenAI Five, prop. n.: A team of five AI players built by OpenAI that learned the team video game Dota 2 by playing against itself, and in April 2019 beat the reigning world-champion human team., Oriol VinyalsOriol Vinyals, prop. n.: A Spanish computer scientist who co-created key neural methods for translation and image captioning, led DeepMind's StarCraft-playing AlphaStar, co-led Google's Gemini models, and in 2026 co-founded the startup Discovery Loop.