OpenAI Five
In plain wordsA team of five AI players built by OpenAI that learned the team video game Dota 2 by playing against itself, and in April 2019 beat the reigning world-champion human team.
A system of five neural-network agents, one per hero, that played the multiplayer battle-arena game Dota 2, trained with proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback. entirely by self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples. at very large scale. Each agent's policy was a recurrent LSTMlong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences. network that read the game state through the game's programmatic interface and acted every few frames across games lasting tens of thousands of steps. A 'team spirit' parameter blended individual and shared rewards, and training ran almost continuously for about ten months on a large cluster of GPUs and CPUs, using 'surgery' techniques to continue training as the network and the game itself changed. The system beat the world-champion team OG 2–0 in April 2019 and then won more than 99 percent of several thousand games against the public. It played a restricted pool of heroes and items, but showed that existing algorithms, scaled up, could cope with long horizons, partial observability and team coordination.
Our agents share a 'team spirit' coefficient, as in OpenAI Five, so they learn to sacrifice for teammates.
Origin
Developed by OpenAI from 2017, after a one-on-one bot beat professional players at The International 2017. The five-player system lost two exhibition games to professional teams at The International 2018 before beating OG at the OpenAI Five Finals in April 2019; Christopher Berner and colleagues described it in 'Dota 2 with Large Scale Deep Reinforcement Learning' (December 2019).
Related
- self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.
- proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
- multi-agent reinforcement learningmulti-agent reinforcement learning, n.: Reinforcement learning with several AI agents learning at once in a shared world, where each must adapt to the others — as teammates, rivals or both.
- deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images.
- AlphaStarAlphaStar, prop. n.: A DeepMind AI that learned to play the real-time strategy game StarCraft II well enough to reach its highest public rank, Grandmaster, in 2019.
Sources
- Berner, C. et al. (2019). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv:1912.06680.
Mentioned in
AlphaStarAlphaStar, prop. n.: A DeepMind AI that learned to play the real-time strategy game StarCraft II well enough to reach its highest public rank, Grandmaster, in 2019.