shard theory

A research program and theory of value formation holding that reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. in brains and in deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract. systems yields not one objective but a collection of 'shards': learned circuits, each strengthened by rewards received in some kind of situation, that switch on when a similar situation recurs and push behavior toward what was rewarded there — a child often praised for tidying might, for instance, acquire a tidying shard that activates in messy rooms. What the agent values is whatever emerges from its shards competing and compromising. Shard theorists argue that reward shapes cognition rather than becoming the agent's goal — Alex Turner's slogan is 'reward is not the optimization target' — so that worries about agents single-mindedly maximizing reward or one mesa-objectivemesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for. are misplaced, and alignment may be achievable by shaping which shards form. Critics ask what testable predictions the theory makes and whether shards would stay stable in highly capable, reflective systems.

On a shard-theory view, the model doesn't want reward; it has learned urges that reward happened to reinforce.