loss curve
In plain wordsA graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong.
A chart that tracks how a model's lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. falls, or fails to fall, as training proceeds — measured in steps, tokens seen or training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules. — usually drawn for both the training data and a held-out validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one.. In large-model pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., where most data are seen only once, training and validation curves nearly coincide, and the curve is often read on log–log axes, where steady improvement looks close to a straight line (power lawpower law, n.: A rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line.). Engineers watch it for divergence, loss spikesloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy. and plateaus, compare runs at equal tokens or FLOP, and fit scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. to it. Its shape also reflects the learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in.: under a warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over. the loss falls slowly in the stable phase and then drops sharply during decay, so mid-run values understate final quality. Per-batch loss is noisy, so curves are usually smoothed with a moving average.
The loss curve flattened at 1.9 until the decay phase, then dropped to 1.75.
Origin
Learning curves date back to Hermann Ebbinghaus's memory experiments (1885). 'Loss curve' became the everyday deep-learning term as training tools logged loss continuously, from TensorBoard (2015) to experiment trackers such as Weights & Biases.
Usage note
Often used interchangeably with learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help., which in classical machine learning more often plots performance against training-set size. A falling training loss with rising validation loss signals overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding..
Related
- loss spikeloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy.
- learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months.
- warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.
- validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one.
- learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help.
Mentioned in
convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help., training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months., loss spikeloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy.