scaling hypothesis

The conjecture that scale is the main ingredient still missing from AI: adding parameters, data and training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules. to an existing general-purpose architecture such as the transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., with only incremental engineering changes, will keep yielding more general capabilities, up to and beyond human level. In Gwern Branwen's formulation, the strong version holds that scale alone makes sophisticated intelligence emerge, because that is the easiest way for a model to reduce prediction error across diverse data; a weak version, which he attributed to DeepMind, holds that scale is necessary but new algorithms must still be found. Supporters cite smooth scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. and the jumps from GPT-2GPT-2, prop. n.: A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous. to GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. and GPT-4GPT-4, prop. n.: OpenAI's 2023 language model, a big step up from the one behind the first ChatGPT. It scored well on many professional and academic exams and could also read images.. Critics such as Gary Marcus and Yann LeCun argue that next-token prediction cannot by itself supply reliable reasoning or world modelsworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.. Since 2024 the debate has shifted toward whether progress now comes mainly from scaling reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. and test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter. rather than pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses..

If the scaling hypothesis is right, the next order of magnitude of compute matters more than any new architecture.