test-time compute
In plain wordsThe computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
Computation spent at inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it. time on a given input, and by extension the strategy of improving outputs by increasing it rather than by training a bigger model. Methods are parallel — sampling many candidate answers and choosing by majority vote (self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts.) or with a verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded. or reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. (best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.) — sequential, generating longer reasoning tracesreasoning trace, n.: The written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see. that revise themselves, or structured search such as tree of thoughtstree of thoughts, n.: A method in which an AI explores several possible next steps of a problem, judges which look promising and backs out of dead ends, like a chess player weighing moves, instead of reasoning in one straight line. or Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average.. On many reasoning tasks accuracy rises roughly linearly with the logarithm of test-time compute, a relationship called test-time or inference scaling; gains are largest where answers can be checked and smallest for knowledge the model lacks. Spending compute per query rather than once in training changes pricing, energy use and the assumptions of compute-based governance.
Doubling test-time compute bought us more accuracy than the next model size up would have.
Origin
Trading search at play time against training was familiar from game AI: AlphaGoAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it. relied on tree search, and Andy Jones's 2021 experiments on the board game Hex quantified the exchange rate between train-time and test-time compute. The idea moved to the center of LLM research in 2024, when Charlie Snell and colleagues (August) showed a smaller model with optimally allocated test-time compute could beat one 14 times larger, and OpenAI's o1 (September) reported gains from time spent thinking.
Usage note
Not to be confused with test-time trainingtest-time training, n.: Letting an AI model briefly update itself on each new problem it is given, learning from that problem's own clues before answering, instead of keeping its knowledge frozen after training., which updates weights on each test input. 'Inference compute' in economic discussions usually means all serving compute, not extra reasoning per query.
Related
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
- thinking budgetthinking budget, n.: A limit, set by whoever uses an AI model, on how much private step-by-step reasoning it may do before answering: more thinking usually means better answers but higher cost and slower replies.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts.
- best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
- training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules.
- test-time trainingtest-time training, n.: Letting an AI model briefly update itself on each new problem it is given, learning from that problem's own clues before answering, instead of keeping its knowledge frozen after training.
Sources
- Snell, C., Lee, J., Xu, K. & Kumar, A. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv:2408.03314 (ICLR 2025). arxiv.org/abs/2408.03314v1
- Jones, A. L. (2021). Scaling Scaling Laws with Board Games. arXiv:2104.03113. arxiv.org/abs/2104.03113
- OpenAI (2024). Learning to Reason with LLMs. OpenAI blog. openai.com/index/learning-to-reason-with-llms
Mentioned in
ARC-AGIARC-AGI, prop. n.: Puzzles made of colored grids — and, since 2026, small video-game-like worlds — that people solve easily but AI long found hard, designed to test whether a machine can learn a new skill from just a few examples., computecompute, n.: The raw computing power used to train and run AI, usually measured by how many specialized chips are used and for how long. With data and algorithms, it is a main driver of AI progress., reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning., scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers., BrowseCompBrowseComp, prop. n.: A test of AI web-research agents: 1,266 puzzle-like questions whose answers take a long, persistent online hunt to uncover, though each answer is short and simple to confirm., compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees., compute governancecompute governance, n.: Using control over the computer chips and data centers that AI depends on — who can buy them, where they go, how they are used — as a way to oversee powerful AI., compute thresholdcompute threshold, n.: A legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models., data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up., inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it., model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., OpenAI o1OpenAI o1, prop. n.: OpenAI's first reasoning model, from 2024. Before replying it writes out a long, hidden series of steps, trying ideas and checking them, which lifted its scores on hard math, science and coding problems far above earlier chatbots'., reasoningreasoning, n.: Working out conclusions step by step from facts, rules or evidence, as in solving a math problem or planning a trip. Whether today's AI truly reasons, or imitates reasoning, is debated., RL scalingRL scaling, n.: The practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably., scaling hypothesisscaling hypothesis, n.: The bet that today's AI methods, simply made bigger with more computing power and data, will keep getting smarter — possibly all the way to human-level intelligence and beyond — without needing fundamentally new ideas., scaling wallscaling wall, n.: The disputed idea that making AI models bigger is running out of steam, so progress will soon stall; 'hitting a wall' became the skeptics' slogan and 'there is no wall' the reply., self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts., System 1 and System 2System 1 and System 2, phr.: A popular way of describing two modes of thinking: fast, automatic and intuitive (System 1) versus slow, effortful and deliberate (System 2). AI researchers borrow it to contrast instant answers with step-by-step reasoning., thinking budgetthinking budget, n.: A limit, set by whoever uses an AI model, on how much private step-by-step reasoning it may do before answering: more thinking usually means better answers but higher cost and slower replies., training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules., AI co-scientistAI co-scientist, n.: An AI system built to work alongside human scientists, proposing and debating new research ideas and hypotheses that people then test in the lab., extended thinkingextended thinking, n.: A setting in some AI models, notably Anthropic's Claude, that lets the model write out private step-by-step reasoning before its final answer, improving results on hard problems., generative reward modelgenerative reward model, n.: A scoring model that judges an AI's answer by writing out its reasoning in words before giving a verdict, instead of simply outputting a number.