hardware lottery
In plain wordsThe idea that research ideas succeed partly by luck — because they happen to suit the computer chips available at the time — rather than purely because they are better.
The situation, named by Sara Hooker, in which a research approach comes out ahead mainly because it suits the chips and software already at hand rather than because it beats its rivals on merit, while ideas that suit the available tools poorly can look like dead ends and be set aside. The central example is deep learning itself: neural networksneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. languished for decades until GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time., built for video games, made dense matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing. cheap. The concept warns that today's specialized acceleratorsAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., tuned to transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. and dense linear algebra, may entrench current methods and penalize alternatives such as sparse, dynamic or neuromorphicneuromorphic computing, n.: Computer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy. approaches, and it argues for hardware, software and algorithm research that keeps more options open.
Origin
Coined by Sara Hooker, then at Google Brain, in the essay 'The Hardware Lottery' (arXiv, September 2020), later published in Communications of the ACM (2021).
Usage note
Often discussed alongside Rich Sutton's bitter lessonthe bitter lesson, phr.: The argument, made by AI pioneer Rich Sutton in 2019, that AI has advanced mostly through general methods that use ever more computing power, not by programming in human knowledge.: methods that scale with compute tend to win, but which methods can scale depends on the hardware available.
Related
- the bitter lessonthe bitter lesson, phr.: The argument, made by AI pioneer Rich Sutton in 2019, that AI has advanced mostly through general methods that use ever more computing power, not by programming in human knowledge.
- GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
- AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
- Huang's lawHuang's law, n.: A claim, named after Nvidia's chief executive, that AI chip performance improves much faster than Moore's law because chips, software and number formats all improve together.
- neuromorphic computingneuromorphic computing, n.: Computer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy.
- deep learning revolutiondeep learning revolution, n.: The rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems.
Sources
- Hooker, S. (2020). The Hardware Lottery. arXiv:2009.06489; Communications of the ACM 64(12), 2021. arxiv.org/abs/2009.06489
Mentioned in
GPGPUGPGPU, abbr.: Using a graphics chip for ordinary number-crunching instead of drawing pictures — the trick that, years later, made modern AI training practical., Huang's lawHuang's law, n.: A claim, named after Nvidia's chief executive, that AI chip performance improves much faster than Moore's law because chips, software and number formats all improve together., neuromorphic computingneuromorphic computing, n.: Computer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy.