nondeterminism
In plain wordsThe fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences.
Variation in a model's output across runs with identical input and settings. Much of it is intended: sampling at nonzero temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone. draws tokens at random, and fixing a random seedrandom seed, n.: A number that fixes the 'random' choices a computer makes; reuse the same seed and settings, and an image generator produces the same picture again. makes it repeatable only if everything else is fixed too. But outputs also vary at temperature 0, because floating-point addition is not associative — results depend on the order in which numbers are summed — and that order can change with GPU kernels, hardware, parallelism and, in served models, batch size. Horace He of Thinking Machines Lab argued in September 2025 that the main culprit in LLM serving is a lack of 'batch invariance': kernels whose numerics change with the number of concurrent requests. In his test, 1,000 temperature-0 runs of Qwen3-235B gave 80 distinct completions, but identical ones with batch-invariant kernels. Nondeterminism complicates debugging, evaluation, caching and on-policy reinforcement learning.
Even at temperature 0 the endpoint's answers drifted, so we pinned a batch-invariant build for the regression suite.
Origin
Nondeterministic GPU computation troubled deep-learning reproducibility from at least the mid-2010s, prompting deterministic modes in frameworks. For LLMs the issue drew attention when developers found that temperature 0 did not give identical outputs; Thinking Machines Lab's 'Defeating Nondeterminism in LLM Inference' (September 2025) offered a widely cited diagnosis.
Usage note
Deliberate randomness from sampling (stochasticstochastic, adj.: Involving chance, so the same starting point can lead to different outcomes. AI chatbots are stochastic: ask the same question twice and you may get two different answers. generation) differs from unintended numerical nondeterminism. Some APIs offer a seed parameter for only 'best-effort' reproducibility.
Related
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- random seedrandom seed, n.: A number that fixes the 'random' choices a computer makes; reuse the same seed and settings, and an image generator produces the same picture again.
- floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
- greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.
- batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
- continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.
Sources
- He, H. & Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism. thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference
Mentioned in
temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text., Mira MuratiMira Murati, prop. n.: An Albanian-born American engineer who, as OpenAI's chief technology officer, oversaw the launch of ChatGPT, and who founded the AI company Thinking Machines Lab in 2025., random seedrandom seed, n.: A number that fixes the 'random' choices a computer makes; reuse the same seed and settings, and an image generator produces the same picture again., reproducibilityreproducibility, n.: Whether other people, or the same people later, can get the same results by rerunning an AI experiment. Many published AI results have proved hard to reproduce, which makes real progress harder to judge., stochasticstochastic, adj.: Involving chance, so the same starting point can lead to different outcomes. AI chatbots are stochastic: ask the same question twice and you may get two different answers., silent data corruptionsilent data corruption, n.: When computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices.