evolution strategies
In plain wordsA family of evolution-inspired methods for tuning a list of numbers: try many random variations of the current best settings, keep those that work better, and adjust how big the variations are as you go.
A class of evolutionary algorithmsevolutionary computation, n.: Problem-solving by simulated evolution, much as breeders improve crops: keep many candidate answers, let the better ones produce slightly altered offspring, drop the worse, and repeat for many rounds. for optimizing real-valued parameter vectors, in which offspring are generated by adding Gaussian noise to parent vectors and the best are selected for the next generation. Notation such as (1+1), (μ, λ) and (μ+λ) states how many parents and offspring there are and whether parents compete with their offspring. A distinctive feature is self-adaptation of the mutation distribution, from Ingo Rechenberg's one-fifth success rule to CMA-ES (covariance matrix adaptation), which learns a full covariance matrix and is a standard method for hard black-box optimization in tens to hundreds of dimensions. Natural evolution strategies interpret the update as gradient ascent on expected fitness under the search distribution. In 2017 researchers at OpenAI showed that ES, estimating gradients from many random perturbations evaluated in parallel, could train neural-network policies competitively with reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time..
Origin
Developed from 1964 by Ingo Rechenberg, Hans-Paul Schwefel and Peter Bienert at the Technical University of Berlin to optimize aerodynamic shapes and nozzles experimentally; Rechenberg's Evolutionsstrategie (1973) set out the theory. Nikolaus Hansen and Andreas Ostermeier introduced CMA-ES in 1996, and Tim Salimans and colleagues applied ES to deep reinforcement learning in 2017.
Related
- evolutionary computationevolutionary computation, n.: Problem-solving by simulated evolution, much as breeders improve crops: keep many candidate answers, let the better ones produce slightly altered offspring, drop the worse, and repeat for many rounds.
- genetic algorithmgenetic algorithm, n.: A problem-solving method that copies natural evolution: many candidate solutions compete, the best are 'bred' by mixing and randomly tweaking them, and over many generations the solutions improve.
- neuroevolutionneuroevolution, n.: Training or designing neural networks by simulated evolution — breeding and mutating whole networks and keeping the best performers — instead of, or alongside, the usual gradient-based training.
- reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.
- Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
- simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops.
Sources
- Rechenberg, I. (1973). Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Frommann-Holzboog.
- Hansen, N. & Ostermeier, A. (2001). Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation 9(2), 159–195.
- Salimans, T., Ho, J., Chen, X., Sidor, S. & Sutskever, I. (2017). Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv preprint.
Mentioned in
evolutionary computationevolutionary computation, n.: Problem-solving by simulated evolution, much as breeders improve crops: keep many candidate answers, let the better ones produce slightly altered offspring, drop the worse, and repeat for many rounds., genetic algorithmgenetic algorithm, n.: A problem-solving method that copies natural evolution: many candidate solutions compete, the best are 'bred' by mixing and randomly tweaking them, and over many generations the solutions improve., neuroevolutionneuroevolution, n.: Training or designing neural networks by simulated evolution — breeding and mutating whole networks and keeping the best performers — instead of, or alongside, the usual gradient-based training.