test-time training

Adaptation

Updating some or all of a model's parameters at inference time, separately for each test input or small batch, using a loss that needs no ground-truth label, then making the prediction with the adapted parameters, which are usually discarded afterward. The original method attached a self-supervised auxiliary task to a shared feature extractor and took gradient steps on that task for each unlabeled test image, improving robustness under distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., test-time training fine-tunes temporarily, often with LoRAlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves., on data derived from the prompt — for instance the demonstration pairs of a few-shot puzzle, expanded by transformations — which in 2024, combined with program synthesis, lifted accuracy on the public ARC-AGIARC-AGI, prop. n.: Puzzles made of colored grids — and, since 2026, small video-game-like worlds — that people solve easily but AI long found hard, designed to test whether a machine can learn a new skill from just a few examples. validation set to what its authors reported as roughly average human level. It trades extra inference computation for adaptability, complementing in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round..

Test-time training on each puzzle's three examples, plus rotations and flips of them, more than doubled our solve rate.

Sequence modeling

A family of sequence-model layers (TTT layers) whose hidden state is itself a small model — a linear map or a two-layer MLP — updated by a step of self-supervised learning on each incoming token, so the layer keeps learning as it reads. Proposed as linear-time alternatives to attention for long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., they are related to linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words. and state space modelsstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer..