How Does OpenAI's o1 Teach LLMs to Reason Better?

TL;DR

OpenAI's o1 models are trained with reinforcement learning to think for longer before answering, applying system-two style reasoning. Unlike AlphaGo's domain-specific search, o1's extended thinking generalizes across many reasoning domains through the text interface. Reasoning helps most on problems where verifying a solution is easier than generating one, like Sudoku.

Transcript

one way to think about reasoning is there are some problems that benefit from from being able to think about it for longer you know there's this classic notion of system one versus system two thinking in humans system one is the more automatic instinctive response and system two is the slower um you know more process driven response um and for some... Read More

Key Insights

• Reasoning is best understood as problems that benefit from thinking longer, drawing on the system-one versus system-two distinction where system one is automatic and instinctive while system two is slower and more process-driven.
• Some tasks gain nothing from extra thinking time. Asking for the capital of Bhutan does not improve with more time, whereas a Sudoku puzzle can be solved by exploring possibilities because a correct solution is easy to recognize.
• The generator-verifier gap defines where reasoning helps most. Problems sit on a spectrum from easy to verify relative to generation, like Sudoku, to as hard to verify as to generate, like naming a capital.
• o1 is OpenAI's first major foray into general inference-time compute, trained with reinforcement learning to think, and is described as fundamentally different from what people are used to with standard LLMs.
• AlphaGo benefited from thinking longer, taking about 30 seconds per move. Forced to act instantly it was noticeably worse than top humans, showing extra thinking time was central to its strength.
• AlphaGo used Monte Carlo tree search, a reasoning method that worked for Go but does not transfer to games like poker, so its thinking approach stayed domain-specific even though its neural network was general.
• o1's key advance is that its way of thinking longer is itself general and works across many domains, unlike prior systems where only the system-one neural component was general.
• Researchers and MDs have used o1 as a brainstorming partner, running ideas past it about gene discovery and gene therapy applications drawn from years of cancer research experience.

Questions & Answers

Q: What is reasoning in the context of language models?

One way to think about reasoning is that there are some problems that benefit from being able to think about them for longer. This connects to the classic notion of system one versus system two thinking in humans, where system one is the more automatic, instinctive response and system two is the slower, more process-driven response. Reasoning covers the kinds of problems where there is a benefit from considering more options and thinking for longer.

Q: Why does more thinking time help on some problems but not others?

For some tasks you do not really benefit from more thinking time. If asked the capital of Bhutan, you could think for two years and it would not raise your accuracy. But for problems like a Sudoku puzzle, you could go through many possibilities for the solution, and because it is easy to recognize a correct solution, given enough time you would eventually figure it out. The benefit depends on the problem type.

Q: What is the generator-verifier gap?

The generator-verifier gap describes situations where it is really hard to generate a correct solution but much easier to recognize when you have one. Problems exist on a spectrum from easy to verify relative to generation, like a Sudoku puzzle, to just as hard to verify as it is to generate a solution, like naming the capital of Bhutan. Reasoning tends to help most where verification is easier than generation.

Q: How is o1 trained and how does it differ from standard LLMs?

The o-series models are trained with reinforcement learning to be able to think, which you could also call reasoning. The team describes this as fundamentally different from what people are used to with LLMs. It has generalized to a lot of different reasoning domains, and OpenAI is excited about this paradigm shift with the new model family. o1 represents OpenAI's first major foray into general inference-time compute.

Q: What are the lessons from AlphaGo that apply to o1?

AlphaGo clearly benefited from thinking longer, taking about 30 seconds to make a move. If it acted instantly it was noticeably worse than top humans, showing extra thinking time mattered. However, AlphaGo ran Monte Carlo tree search, a form of reasoning that worked well for Go but does not work in a game like poker. So its thinking methods stayed domain-specific even though the underlying neural network was general.

Q: Why is o1's approach to reasoning considered general?

For earlier systems, the methods for thinking longer were specific to particular domains, even though the system-one neural network component behind them was very general. What is notable about o1 is that its way of thinking for longer is itself quite general and can be used across many different domains. This generality is being demonstrated by giving o1 to users and seeing the range of tasks they accomplish with it.

Q: Did the OpenAI researchers always have conviction o1 would work?

Conviction varied among the team. Noam felt there was conviction that something in this direction was promising, though the exact path was never clear and much prior research did not pan out. Hunter had less conviction initially, with his aha moment coming when he read model outputs approaching problems differently. Ilge, at OpenAI five and a half years, originally thought robotics and embodied AI were the way forward and found the reasoning path not obvious for a long time.

Q: How are people using o1 in unexpected ways?

One surprising use the team highlighted is researchers and MDs using o1 as a brainstorming partner. People who have been in cancer research for many years have been running ideas past the model about what they can do with gene discovery and gene therapy type applications. OpenAI also values iterative deployment, releasing the technology so they can safely see how the world interacts with it and learn what these tools are useful for.

Summary & Key Takeaways

• Reasoning refers to problems that benefit from longer thinking, echoing the system-one versus system-two framework. System one is automatic and instinctive; system two is slower and process-driven. Some questions, like a country's capital, gain nothing from extra time, while puzzles like Sudoku reward extended search because correct solutions are easy to recognize.

• o1, also called Project Strawberry, is OpenAI's first major foray into general inference-time compute. The o-series models are trained with reinforcement learning to think and reason, which the team describes as fundamentally different from standard LLMs. The approach has generalized across many reasoning domains, marking a paradigm shift for the model family.

• AlphaGo showed the value of thinking time, taking roughly 30 seconds per move and performing worse when forced to act instantly. But its Monte Carlo tree search only worked for Go, not poker. o1's advance is that its extended-thinking process is itself general, working across domains through the flexible text interface.