power law
In plain wordsA rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line.
- 1
Scaling
A relationship of the form y = a·xᵏ, whose signature is scale invariance: scaling x by a factor c always scales y by cᵏ, so log y is a linear function of log x with slope k. In AI the canonical case is loss falling as a power of parameters, data or compute (k < 0), usually plus a constant for the irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence.; the exponent says what a tenfold increase buys — with k = −0.05, each tenfold increase in compute cuts the reducible loss by about 11%. Power laws are scale-free, with no characteristic size, which is why trends fitted on small runstraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months. can be extrapolated, and why small errors in a fitted exponent grow into large differences many orders of magnitude out.
Validation loss follows a power law in compute with an exponent near −0.05.
- 2
Probability
A heavy-tailed probability distribution whose density or frequency falls off as x^−α, as in Zipf's law for word frequencies and Pareto distributions of income. Such distributions shape language data and tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. statistics, and produce the long taillong tail, n.: The huge number of rare things — uncommon words, unusual road situations, obscure facts — that each appear seldom but together make up a large share of what an AI system meets. of rare facts and phrases that models learn slowly.
Origin
Power-law relations are old in science (Kepler's third law, 1619). Vilfredo Pareto described power-law income distributions in the 1890s and George Kingsley Zipf word-frequency laws in the 1930s–1940s. In machine learning, power-law learning curves were studied from the early 1990s and became central with neural scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. (2017–2020).
Usage note
A straight line over one or two decades on a log–log plot is weak evidence of a power law, and real curves can bend or break. Do not confuse with exponential growth, which is straight on a plot with only the vertical axis logarithmic.
Related
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence.
- long taillong tail, n.: The huge number of rare things — uncommon words, unusual road situations, obscure facts — that each appear seldom but together make up a large share of what an AI system meets.
- Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.
- training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules.
Mentioned in
scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., Jared KaplanJared Kaplan, prop. n.: An American physicist who showed that language models improve predictably as they grow and train on more data, and who co-founded the AI company Anthropic., learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help., long taillong tail, n.: The huge number of rare things — uncommon words, unusual road situations, obscure facts — that each appear seldom but together make up a large share of what an AI system meets., loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong., generative recommendationgenerative recommendation, n.: A newer approach to recommendation that uses language-model-style AI to predict the next item a person will want, one step at a time, much as a chatbot predicts the next word., irreducible lossirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence., Zipf's lawZipf's law, n.: The pattern that word use is lopsided: in a big text the top word is roughly twice as common as the runner-up and ten times as common as the tenth, so a few words dominate and most are rare.