von Neumann bottleneck
In plain wordsThe traffic jam caused by ordinary computers keeping data and processing in separate places, so information must constantly shuttle back and forth through one narrow channel.
The limit on performance and energy efficiency imposed by the stored-program (von Neumann) architecture, in which a processor and a separate memory exchange instructions and data over one channel, so that the processor spends much of its time and energy waiting for, and moving, data. Caches, wider buses and high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough. ease but do not remove it. For neural networksneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., whose weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. must be read for every input, the bottleneck motivates non-von Neumann designs that compute where data are stored — analog in-memory computinganalog in-memory computing, n.: Doing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise., processing-near-memory and neuromorphicneuromorphic computing, n.: Computer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy. chips — and dataflow accelerators that keep operands in local SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it..
Origin
Coined by John Backus in his 1977 ACM Turing Award lecture, 'Can Programming Be Liberated from the von Neumann Style?' (published in Communications of the ACM, August 1978), which criticized both the hardware channel and the word-at-a-time programming style it imposed. The architecture traces to John von Neumann's 'First Draft of a Report on the EDVAC' (1945).
Usage note
Backus's main target was programming languages; hardware designers later adopted the phrase in its narrower physical sense. Compare memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive., a quantitative trend rather than an architectural feature.
Related
- memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.
- analog in-memory computinganalog in-memory computing, n.: Doing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise.
- neuromorphic computingneuromorphic computing, n.: Computer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy.
- memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.
- CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math.
Sources
- Backus, J. (1978). Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs. Communications of the ACM 21(8).
Mentioned in
CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math., analog in-memory computinganalog in-memory computing, n.: Doing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise., memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive., neuromorphic computingneuromorphic computing, n.: Computer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy.