safe reinforcement learning

Reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. in which avoiding unacceptable outcomes counts alongside returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible., whether the danger lies in the exploratory actions taken while learning or in the behavior of the finished policy. A common formalization is the constrained Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice., in which the agent maximizes reward subject to limits on expected cumulative costs such as collisions, energy use or rule violations, solved with Lagrangian methods or constrained policy optimization. Other strands optimize risk-sensitive criteria, such as conditional value at risk, instead of the mean; restrict exploration to actions certified safe by a model, a 'shield' or a control-barrier function (safe exploration); learn conservatively from logged data (offline reinforcement learningoffline reinforcement learning, n.: Learning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning.); and keep humans able to intervene. Benchmarks such as OpenAI's Safety Gym (2019) and its successors measure the trade-off between reward and cost. The field overlaps with AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable. concerns such as reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant. and safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them., which arise when the reward itself fails to encode what is unsafe.

We framed the warehouse robot as a constrained MDP: maximize throughput while keeping expected collisions below one per thousand hours.