compute-optimal training

Training in which model size N and training-token count D are chosen to minimize final loss for a given compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees. C, subject to C ≈ 6ND. A larger model learns more per token but can process fewer tokens for the same budget, so loss plotted against N at fixed C is U-shaped, and the bottom of the U is the compute-optimal size; IsoFLOP analysisIsoFLOP analysis, n.: A way of finding the best model size for a given amount of computing: train several models of different sizes at exactly the same computing cost and see which one ends up best. locates it empirically. For dense transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. on web text the Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model. put the optimum near 20 tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size., with N and D each growing roughly as the square root of C; the ratio shifts with data quality, architecture, tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., sparsity and data repetition. Compute-optimality ignores the cost of serving the model, which is why production models are usually overtrainedovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards., and it assumes a learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in. matched to each run's length.

A compute-optimal 7B model would see about 140 billion tokens; we trained ours on two trillion.