model capacity

The range and complexity of functions that a model class can represent or fit, and hence how much information it can absorb from data. Formal measures from learning theory include the VC dimensionVC dimension, n.: A measure of how flexible a family of prediction rules is: the largest number of points it can label in every possible way. More flexible families need more data to learn reliably., Rademacher complexity and description length; parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them. is a crude proxy. Ian Goodfellow, Yoshua Bengio and Aaron Courville distinguish representational capacity, what an architecture could express, from effective capacity, what a given optimizer and regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. actually reach. Classical theory ties capacity to the bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.: too little causes underfittingunderfitting, n.: When a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from., too much overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.. Deep networks complicate the picture: they can fit randomly labeled data yet generalize on real data, and test error can fall again beyond the point where the model interpolates its training set (double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.). For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., capacity is also measured as storable information: Zeyuan Allen-Zhu and Yuanzhi Li (2024) estimated about 2 bits of factual knowledge per parameter, and John Morris and colleagues (2025) about 3.6 bits of memorized data per parameter for GPT-style models.

Training and validation loss plateaued together at a high value, so we judged the model capacity-limited and doubled its width.