Q*

OpenAI project

The reported code name of an internal OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation. research effort that became public in November 2023, days after the board's brief removal of chief executive Sam AltmanSam Altman, prop. n.: An American entrepreneur who co-founded OpenAI and runs it; the company's 2022 release of ChatGPT turned him into one of the public faces of the AI era.. Reuters reported that some staff had written to the board warning of a powerful discovery, citing a model called Q* that could solve grade-school mathematics problems it had not seen before, and that the letter was one of several factors behind the firing; The Verge, citing a person familiar with the matter, reported that the board never received such a letter. The Information described Q* as building on work by Ilya SutskeverIlya Sutskever, prop. n.: A Russian-born Israeli-Canadian computer scientist who co-built the AlexNet image recognizer, helped found OpenAI and lead its research until 2024, and now heads the startup Safe Superintelligence Inc. and developed by Jakub Pachocki and Szymon Sidor, using computer-generated data and extra computation to improve reasoning. According to Reuters, OpenAI acknowledged the project in an internal message to staff, but it has never published a technical account; Altman called the reports an 'unfortunate leak'. In July 2024 Reuters reported that the work continued under the code name Strawberry, which was widely identified with OpenAI o1OpenAI o1, prop. n.: OpenAI's first reasoning model, from 2024. Before replying it writes out a long, hidden series of steps, trying ideas and checking them, which lifted its scores on hard math, science and coding problems far above earlier chatbots'. on its release in September 2024.

In hindsight, the Q* leak was the first public hint that OpenAI was betting on reasoning at inference time.

Reinforcement learning

Standard notation for the optimal action-value function: the expected discounted return from taking action a in state s and acting optimally afterward. It satisfies the Bellman optimality equationBellman equation, n.: A rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time., and Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next. estimates it from experience; an agent that knows Q* acts optimally by always choosing the action with the highest value.