maximal update parametrization
In plain wordsA recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one.
A rule for how a network's initialization variances, layer multipliers and per-layer learning rateslearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. should scale with width so that every layer's activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. change by a width-independent amount per update — the largest change that stays stable — and features keep being learned as width grows. Under standard parametrization, optimal learning rates and other hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data. drift as models widen, and in the neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior. limit feature learning vanishes; under μP the infinite-width limit still learns features, and many optimal hyperparameters stay approximately fixed across widths. Because those values hold steady, they can be searched for on a small proxy model and carried over unchanged to the full model, a procedure its authors call μTransfer. In the 2022 demonstration, hyperparameters found on a 40-million-parameter proxy yielded a 6.7-billion-parameter model that beat the published GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. results for that size, for a tuning bill of roughly 7% of the full pretraining cost. Later work extends the approach to depth.
Under μP the optimal learning rate from the 50M proxy carried over to the 7B run.
Origin
Introduced by Greg Yang and Edward Hu at Microsoft Research in 'Feature Learning in Infinite-Width Neural Networks' (Tensor Programs IV; arXiv November 2020, ICML 2021). μTransfer was demonstrated by Yang, Hu, Igor Babuschkin and colleagues from Microsoft and OpenAI in Tensor Programs V (March 2022); Cerebras-GPT and MiniCPM are documented adopters.
Usage note
μP fixes how hyperparameters scale with width; it does not choose them, and batch size, training length and weight decay need their own scaling rules. Its authors spell it 'parametrization'; 'parameterization' is also common.
Related
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.
- weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers.
- neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.
- training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
Sources
- Yang, G. & Hu, E. J. (2021). Feature Learning in Infinite-Width Neural Networks. ICML 2021. arXiv:2011.14522. arxiv.org/abs/2011.14522
- Yang, G., Hu, E. J., Babuschkin, I., Sidor, S. et al. (2022). Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. arXiv:2203.03466. arxiv.org/abs/2203.03466v2
Mentioned in
hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it., weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.