optimal control

The branch of control theorycontrol theory, n.: The mathematics of making machines behave as intended by constantly measuring what they are doing and correcting them, as a thermostat, cruise control or self-balancing robot does. that seeks control inputs, as functions of time or of state, that minimize a cost functional — typically a running cost on state and effort plus a terminal cost — subject to the system's dynamics and constraints. Its two classical foundations are Pontryagin's maximum principle, which gives necessary conditions for an optimal trajectory, and Bellman's dynamic programmingdynamic programming, n.: A problem-solving method that breaks a big problem into smaller overlapping pieces, solves each piece once and reuses the answers — the basis for computing the best long-term strategy step by step., which characterizes the optimal feedback law through a value function satisfying the Hamilton–Jacobi–Bellman equation, the continuous-time counterpart of the Bellman equationBellman equation, n.: A rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time. in reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.. The linear-quadratic regulator (LQR), for linear dynamics and quadratic costs, has a closed-form solution through the Riccati equation and is a workhorse of robotics; for nonlinear systems, differential dynamic programming and iterative LQR solve a sequence of local approximations. Trajectory optimizationtrajectory optimization, n.: Calculating the best way for a robot to move over time, choosing exact positions, speeds and forces moment by moment so the motion is smooth, fast or energy-efficient and avoids obstacles. and model predictive controlmodel predictive control, n.: A control method that repeatedly plans the best moves a few seconds ahead using a model of the machine, carries out only the first move, then plans again with fresh measurements. are its main computational offshoots, and reinforcement learning can be viewed as optimal control with unknown or sampled dynamics.

We linearize around the upright pose and use LQR to keep the robot balanced.