learning curve

Performance versus data

A plot of a model's error or accuracy, on training and held-out data, as a function of the amount of training data. Reading it diagnoses the bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.: if training and validation error converge at a high value, the model is underfittingunderfitting, n.: When a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from. and more data will not help, while a large persistent gap indicates overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding. that more data or regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. can reduce. Learning curves of modern models often follow power lawspower law, n.: A rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line. in dataset size, the empirical basis of data scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., and extrapolating them guides decisions about data collection; they can also be non-monotonic (see double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.).

The learning curve was still steep at 50,000 examples, so we funded more labeling.

Performance versus training time

Loosely, a plot of training and validation loss or accuracy against stepstraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps. or epochsepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once. within a single run, used to choose a stopping point, detect overfitting and spot instabilities; for large models this is usually called a loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong..