Richard Bellman

American applied mathematician whose invention of dynamic programmingdynamic programming, n.: A problem-solving method that breaks a big problem into smaller overlapping pieces, solves each piece once and reuses the answers — the basis for computing the best long-term strategy step by step., at the RAND Corporation in the early 1950s, gave optimal control and reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. their mathematical core. His principle of optimality — whatever the first decisions, the remaining decisions of an optimal policy must be optimal from the state they reach — yields the recursive Bellman equationBellman equation, n.: A rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time. for the value of each state, from which value iterationvalue iteration, n.: A step-by-step method for finding the best strategy when all the rules and rewards of a problem are known: repeatedly re-score every situation from the scores of the situations it can lead to, until the scores stop changing., policy iteration and, later, temporal difference learningtemporal difference learning, n.: A way of learning predictions by adjusting each guess toward the next, slightly better-informed guess instead of waiting for the final outcome — like revising a weather forecast every day as the date approaches. and Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next. descend. He formulated sequential decision-making under uncertainty as what are now called Markov decision processesMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice. (1957), coined the phrase curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much. for the explosive growth of computation with the number of state variables, and co-discovered the Bellman–Ford shortest-path algorithm (1958). He moved to the University of Southern California in 1965, wrote more than 600 papers and some 40 books, and received the IEEE Medal of Honor (1979).