scaling wall
In plain wordsThe disputed idea that making AI models bigger is running out of steam, so progress will soon stall; 'hitting a wall' became the skeptics' slogan and 'there is no wall' the reply.
A hypothesized point of sharply diminishing returns in AI progress from scaling — training ever-larger models on more data and compute — and the recurring debate over whether the field has reached it. Skeptics point to the limited supply of high-quality text, rising costs, shrinking gains from pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. scale and persistent failures of reasoning and reliability; opponents argue that progress continues through other levers such as test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter., reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. on reasoning tasks, synthetic data, better harnessesharness, n.: The software around an AI model that gives it instructions and tools, carries out its actions and decides when it stops. The model is the engine; the harness is the rest of the car. and agents, so that any wall blocks one recipe rather than AI as a whole. Because progress can be measured in many ways — benchmark scores, loss curvesscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., economic usefulness — both sides regularly claim vindication, which makes 'the wall' as much a narrative device as an empirical claim.
Every quiet month brings a new round of scaling-wall takes, and every big release a round of 'there is no wall'.
Origin
Gary Marcus's essay 'Deep Learning Is Hitting a Wall' (Nautilus, March 2022) made the phrase a skeptics' banner. It revived in November 2024, when news reports said new frontier models were improving less than hoped from pretraining scale alone; OpenAI's Sam Altman replied on X that there is no wall, and the argument has recurred with each major release.
Usage note
Separate claims about pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. scaling, about particular benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. and about overall capability, which can diverge. See scaling hypothesisscaling hypothesis, n.: The bet that today's AI methods, simply made bigger with more computing power and data, will keep getting smarter — possibly all the way to human-level intelligence and beyond — without needing fundamentally new ideas. for the opposing thesis and AI winterAI winter, n.: A slump in which disappointment over AI's unmet promises leads governments and companies to pull back money and attention. Historians count two big ones: one in the 1970s and one starting in the late 1980s. for historical slowdowns.
Related
- scaling hypothesisscaling hypothesis, n.: The bet that today's AI methods, simply made bigger with more computing power and data, will keep getting smarter — possibly all the way to human-level intelligence and beyond — without needing fundamentally new ideas.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
- AI winterAI winter, n.: A slump in which disappointment over AI's unmet promises leads governments and companies to pull back money and attention. Historians count two big ones: one in the 1970s and one starting in the late 1980s.
- it's over, we're so backit's over, we're so back, phr.: A meme describing the mood swings of the AI world: one day bad news means 'it's over', the next a new model or result means 'we're so back'.
- Gary MarcusGary Marcus, prop. n.: An American cognitive scientist and leading critic of deep learning, who argues that today's AI cannot reason reliably and needs to be combined with symbolic methods.
- scaling hypothesisscaling hypothesis, n.: The bet that today's AI methods, simply made bigger with more computing power and data, will keep getting smarter — possibly all the way to human-level intelligence and beyond — without needing fundamentally new ideas.