parameter
In plain wordsOne of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned.
- 1
Machine learning
A numerical value inside a model that is learned from data during trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better.: chiefly the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. and biasesbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system. of a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., or the coefficients of a linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.. Parameters are distinguished from hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., such as the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. or the number of layers, which are set before training and govern how the parameters are learned. The total, the parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them., is the usual shorthand for model size, from thousands in early networks to hundreds of billions or more in frontier modelsfrontier model, n.: One of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules.. In a mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. model only a fraction of the parameters is used for each token, so total and active counts are reported separately.
The 7-billion-parameter model fits on a single GPU once quantized to four bits.
- 2
Statistics
A numerical characteristic of a population or probability distribution, such as a mean, a variance or a regression coefficient, that is typically unknown and estimated from a sample. A parametric modelparametric model, n.: A model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones. has a fixed, finite set of parameters; a nonparametric modelnonparametric model, n.: A model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example. lets its complexity grow with the data.
- 3
Software and APIs
An input to a function or a configurable setting. Model APIsmodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves. call settings such as temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., maximum output length and top-ptop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%. 'sampling parameters', although they are not learned and are better described as inference-time settings.
Origin
From Greek para ('beside') and metron ('measure'), first a term of geometry. The statistical sense, a constant characterizing a population as opposed to a statistic computed from a sample, was fixed by Ronald Fisher in his 1922 paper on the mathematical foundations of statistics; machine learning extended it to any learned quantity in a model.
Usage note
Popular writing sometimes equates parameters with neurons or brain synapses; the analogy is loose. Parameter count alone does not determine capability: training data, compute and architecture matter at least as much.
Related
- weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself.
- hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.
- parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.
- trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better.
- parametric modelparametric model, n.: A model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones.
- overparameterizationoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
- hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.
Mentioned in
artificial neuronartificial neuron, n.: The basic building block of a neural network: a tiny calculator that multiplies each incoming number by a weight, adds them up, and passes the total through a simple function to decide its output., datadata, n.: Recorded facts, measurements, text, images or sounds that a computer can store and process. AI systems learn their skills from large amounts of data rather than from rules written by hand., FLOPFLOP, n.: A single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage., machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats., modelmodel, n.: A simplified working version of something. In AI it usually means the trained program, a big set of learned numbers, that takes an input and produces an answer, prediction or piece of content., neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system., training settraining set, n.: The examples a machine-learning model learns from, like the worked problems a student practices on, kept separate from the examples later used to check how well it learned., active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer., derivativederivative, n.: How fast something changes as one of its inputs changes — the steepness of a curve at a point. Training an AI means repeatedly following such slopes downhill., full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., learninglearning, n.: Getting better at something through experience. For a machine, it means changing its internal settings based on data so that it performs a task better than before., likelihoodlikelihood, n.: A score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen., maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this., optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment., parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.