Reasoning model

OpenAI and DeepSeek trained their reasoning models with reinforcement learning, in which the model’s outputs are scored by a reward signal and the model adjusts its behavior to raise its reward. Where an answer can be checked by a rule, the check supplies the reward. In DeepSeek’s R1 work, a math answer had to appear in a set format so that it could be checked automatically, and generated code could be checked against predefined test cases. The report on the Tulu 3 language models (Lambert et al., November 2024) calls training with rewards from such automatic checks reinforcement learning with verifiable rewards, or RLVR. As training went on, DeepSeek-R1-Zero wrote longer chains of thought, and its average pass@1 score on the 2024 AIME, a US high school mathematics competition, rose from 15.6% to 77.9%. Pass@1 is the share of sampled answers that are correct.

The extra reasoning costs computation while the model answers, which OpenAI calls test-time compute. In OpenAI’s tests on the 2024 AIME exams, GPT-4o solved an average of 12% of problems, while o1, OpenAI’s reasoning model, averaged 74% with one sample per problem. For example, an agent built on a reasoning model may write hundreds or thousands of tokens of reasoning before each decision, which adds time to every step it takes. OpenAI decided not to show users o1’s raw chain of thought and to show a model-generated summary instead. Where the reasoning is shown, it may not reflect what drove the answer (see chain of thought).

OpenAI introduced o1 in Learning to Reason with LLMs (12 September 2024) as “a new large language model trained with reinforcement learning to perform complex reasoning”. DeepSeek-AI described its method in DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (22 January 2025; published in Nature in 2025) and released the model weights. The paper reports that reasoning abilities can be developed through pure reinforcement learning, without human-labeled examples of reasoning. Anthropic wrote on 3 April 2025 that reasoning models had been everywhere since late the previous year (Reasoning models don’t always say what they think).