IsoFLOP analysis

A method for estimating compute-optimal model size in which, for each of several fixed compute budgetscompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees., models of different parameter counts are trained on token counts chosen so that every run uses the same number of FLOP (C ≈ 6ND). Plotting final loss against model size for one budget gives an IsoFLOP profile, typically U-shaped; its minimum, often located by fitting a parabola in log N, marks the optimal size. Repeating this across budgets and fitting power laws to the minima yields N_opt(C) and D_opt(C) for extrapolation to the full run. It was the second of three methods in the Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model. paper and has been used to size production models, notably Meta's Llama 3 405B. Each run needs a learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in. matched to its own length, which makes sweeps expensive; warmup-stable-decay scheduleswarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over. reduce the cost.