self-improvement

Training

Any training procedure in which a model's own outputs supply most of its new training signal: generating answers and keeping those that pass a filter (STaRself-taught reasoner, n.: A training method in which a language model writes step-by-step explanations for problems, keeps the ones that led to correct answers, and learns from them, so it gradually teaches itself to reason., expert iterationexpert iteration, n.: A learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter., rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work. fine-tuning); labeling unlabeled data with its own confident predictions (classical self-training); judging its own responses to create preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like. (self-rewarding models, reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging.); or playing against copies of itself (self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.). Self-improvement works when an external anchor keeps the signal honest — a verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., ground-truth answers, game rules or a fixed stronger judge. Without one, errors and biases can compound, and repeated training on self-generated data can degrade quality (model collapsemodel collapse, n.: The gradual decline of AI models trained on the output of earlier AI models: rare details vanish first, then quality and variety erode, like a photocopy of a photocopy.). In practice gains often plateau after a few iterations.

Self-improvement stalled after the third round because the model could no longer tell its own best answers apart.

AGI and superintelligence

An AI system's improvement of its own capabilities, including by designing better training methods, code or successor systems; when the process feeds on itself it is called recursive self-improvementrecursive self-improvement, n.: An AI improving its own design or training, so that each improved version is better at making the next improvement. It is the engine usually imagined behind an intelligence explosion., central to intelligence explosionintelligence explosion, n.: The idea that once AI gets good enough at improving AI, each smarter version could build a smarter one faster still, so abilities would snowball far beyond human level in a short time. arguments.