offline reinforcement learning
In plain wordsLearning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning.
Reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. from a fixed, previously collected dataset of transitions, with no further interaction with the environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score. during training. Its appeal is that records already accumulated in driving, healthcare, recommendation or robot fleets could yield decision-making policies in settings where online trial and error is costly or dangerous. The central difficulty is distributional shift: standard off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one. algorithms overestimate the value of actions absent from the data, because no new experience ever corrects those errors, and the learned policy then exploits them. Offline methods therefore stay close to the data: they constrain the policy to the behavior distribution (BCQ, BEAR, TD3+BC), penalize the values of unseen actions (conservative Q-learning), learn only from in-sample actions (implicit Q-learning) or model trajectories directly, as in the decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action.. Benchmarks such as D4RL standardized evaluation. Off-policy evaluation — estimating a policy's value before deploying it — is a companion problem, and offline pretraining followed by online fine-tuning is common practice.
We trained the dispatch policy with offline RL on two years of logs, then tested it in shadow mode.
Origin
Earlier called batch reinforcement learning, with methods such as Damien Ernst and colleagues' tree-based fitted Q iteration (2005) and Martin Riedmiller's neural fitted Q iteration (2005). The name 'offline reinforcement learning' was popularized by Sergey Levine, Aviral Kumar, George Tucker and Justin Fu's tutorial and review (May 2020), which appeared alongside the D4RL benchmark and conservative Q-learning.
Related
- off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one.
- decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action.
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened.
- behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
- on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes.
Sources
- Levine, S., Kumar, A., Tucker, G. & Fu, J. (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv:2005.01643. arxiv.org/abs/2005.01643
- Kumar, A., Zhou, A., Tucker, G. & Levine, S. (2020). Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS 2020.
- Ernst, D., Geurts, P. & Wehenkel, L. (2005). Tree-Based Batch Mode Reinforcement Learning. Journal of Machine Learning Research 6, 503–556.
Mentioned in
experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened., imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one., deadly triaddeadly triad, n.: The risky combination of three common ingredients in reinforcement learning — approximation, learning from guesses, and learning from other strategies' data — which together can make learning spin out of control., decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action., goal-conditioned reinforcement learninggoal-conditioned reinforcement learning, n.: Training one AI agent to reach many different goals — whichever target it is given — instead of a single fixed task, so a robot can, say, place an object anywhere asked., safe reinforcement learningsafe reinforcement learning, n.: Reinforcement learning designed to avoid dangerous or unacceptable behavior — both in what the finished agent does and in the risky experiments it might try while still learning., Sergey LevineSergey Levine, prop. n.: A Berkeley computer scientist, one of the most prolific researchers on how robots can learn skills from data and experience, and a co-founder of the robotics company Physical Intelligence.