neural collapse

A set of geometric regularities that emerge in the last-layer features and classifier of deep networks trained for classification well past zero training error (the 'terminal phase of training'): the features of all training examples in a class collapse toward their class mean; the class means, centered on the global mean, become equidistant and equiangular, forming a simplex equiangular tight frame, the most separated arrangement possible; the classifier's weight vectors align with those means up to scaling; and classification reduces to choosing the nearest class mean. Observed across standard architectures and image datasets, it has been derived theoretically in 'unconstrained features' models that treat last-layer features as free variables, where it is the optimum of regularized cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. or squared loss. Neural collapse bears on robustness, on transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data. (collapsed features can transfer worse) and on class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., where a 'minority collapse' variant merges rare classes; partial versions are reported in intermediate layers and in language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code..