compute budget
In plain wordsThe total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees.
The fixed quantity of computation available for a training project or run, usually expressed in floating-point operationsFLOP, n.: A single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules. (for example 10²⁴ FLOP) or in accelerator-hours (GPU-hoursGPU-hour, n.: One graphics chip working for one hour — the everyday unit for how much computing a job used and for renting AI chips by the hour.), and ultimately bounded by money, hardware and time. In scaling-law analysis the budget C is the constraint under which model size N and training tokens D are chosen to minimize loss; for dense transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. C ≈ 6ND, so at a given budget a larger model must see fewer tokens (see compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed.). Real plans also reserve compute for ablations and proxy runs, evaluation, restarts and post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., and they weigh expected inference demand, which favors smaller, overtrainedovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards. models. By extension the term is used for inference, as in thinking budgetthinking budget, n.: A limit, set by whoever uses an AI model, on how much private step-by-step reasoning it may do before answering: more thinking usually means better answers but higher cost and slower replies. and test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter..
With a 3 × 10²³ FLOP budget, the compute-optimal dense model is roughly 50 billion parameters.
Origin
Generic engineering usage. It became a pivot of AI research through scaling-law papers asking how best to spend a fixed budget: Jared Kaplan and colleagues at OpenAI (2020) and Jordan Hoffmann and colleagues at DeepMind (2022), whose answers differed sharply.
Related
- training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules.
- compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed.
- Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- GPU-hourGPU-hour, n.: One graphics chip working for one hour — the everyday unit for how much computing a job used and for renting AI chips by the hour.
- FLOPFLOP, n.: A single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules.
- test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
Mentioned in
scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model., compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed., frontier labfrontier lab, n.: One of the handful of companies building the most advanced AI models of the moment, spending billions of dollars on computing power to train them., training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules., training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months., IsoFLOP analysisIsoFLOP analysis, n.: A way of finding the best model size for a given amount of computing: train several models of different sizes at exactly the same computing cost and see which one ends up best., YOLO runYOLO run, n.: Slang for launching a big, expensive training run without the usual careful small-scale tests first — changing many things at once and trusting experience and instinct that it will work. YOLO means 'you only live once'.