How to Train Open-Source Agentic RL Models Like o3

TL;DR

Building an open-source general-purpose agent like o3 requires multi-turn reinforcement learning where tools are used in the loop and rewards are based on the final output after a sequence of tool calls. Key unlocks include treating image manipulation and sub-agents as tool calls, using turn-level rewards for credit assignment, and offloading work to smaller models to avoid blowing up context length.

Transcript

Yeah. Okay. Um, today I'm going to be talking to you all a little bit about uh the frontier directions that we're all thinking about in terms of open source agent RL um and what basically sketching what the road map might be towards training something that looks like an 03 model. What does it take to get something that is like a very good general p... Read More

Key Insights

• Multi-turn RL trains a model to use tools in the loop to solve a problem, then rewards it based on the final output after a sequence of tool calls, moving beyond single-turn RLVR models like o1 and R1.
• '10-minute AGI' is a mental model for o3: anything a human can do in 10 minutes, o3 can usually now reason through, and pushing how long models operate is a plausible path to more capable systems.
• Credit assignment becomes very tricky as task length scales, so turn-level rewards may be needed to ensure the model gets credit for individual moves rather than only the final answer across 100 tool calls.
• Multimodal reasoning is best handled by having models use programmatic tools to manipulate images (crop, zoom, rotate) rather than generating images end-to-end in the chain of thought.
• Treating modes of intelligence as tool calls is a powerful mental model, letting you refocus the model's attention the way a human instinctively zooms in to read a license plate.
• Sub-agents can serve as tools whose only job is to answer a question about a website, so the main model avoids holding 100 full websites in its context window and can parallelize work.
• Off-the-shelf small models are already decent LLMs, so reasoning sub-tasks can be offloaded without end-to-end training, and the main model is trained only to prompt them well.
• Async and overlapped execution, centralized or decentralized, hides inefficiency bubbles in the compute structure by running everything in parallel with some lag.

Questions & Answers

Q: What is multi-turn RL and how does it differ from single-turn RL?

Multi-turn RL trains a model with tools in the loop so it learns to use those tools in service of solving a problem, then gives rewards based on the final output after a sequence of tool calls. This moves beyond the single-turn RL world of chatbots and the single-turn RLVR world of models like o1 and R1, which just guess the right answer to one problem. The goal is to make models more agentic by incorporating reinforcement learning across many turns of tool use.

Q: What is meant by '10-minute AGI' in the context of o3?

'10-minute AGI' is a mental model the speaker likes for thinking about o3: anything you can do as a human in 10 minutes, o3 is usually going to be able to reason through now. If you can crack the scaling direction of pushing the boundary of how long these models can go out and do these things, that is a plausible path toward more capable systems. For example, o3 can be given a hard problem and produce a pretty good sketch of a solution after 15 minutes.

Q: Why is credit assignment a challenge when scaling agentic models?

As task length scales, models are no longer doing just one thing; they may perform 100 tool calls, and deep research can go up to 100. This makes the credit assignment problem very tricky. Rather than waiting only for the final answer at the end, you probably want intermediate verification and something like turn-level reward, so the model gets credit for the moves it makes along the way. Ensuring the model is rewarded for good intermediate actions becomes harder the longer the task runs.

Q: How does o3 handle multimodal reasoning with images?

Instead of generating images end-to-end in the chain of thought, it seems more efficient and easier to have models use programmatic tools to manipulate images. With o3 you can crop an image, zoom in, and rotate it. This ability to work with images lets the model understand them in different ways by shifting how they sit in its context window to emphasize certain things more. It mirrors how humans instinctively refocus attention, like zooming in to read a license plate in a geolocation task.

Q: Why do sub-agents help avoid context-length problems?

If a model reads 100 websites and puts every full website into context, the context limit blows up really quickly. Instead, you can use sub-agents whose only job is to answer a question about a website. This lets you parallelize aspects of producing chains and offload reasoning tasks to off-the-shelf small models that are already decent LLMs. You then train only the main model to learn how to prompt these sub-models and use them well, keeping the main reasoning chain compact.

Q: Why don't group-chat multi-agent systems work well?

The speaker notes that systems like the CrewAI approach, where you put a coder agent and a finance agent in a group chat, are largely silly and do not work very well because the models are not trained to do that. They are never trained in a way that actually reinforces those patterns. Multi-agent RL is genuinely hard; the speaker spent five years on it in grad school, and the algorithms for the things you would really want to do do not even exist yet.

Q: How can smaller models be used as tools without end-to-end training?

Many reasoning tasks can be offloaded to any good-enough off-the-shelf LLM because these small models are already decent at being an LLM. For tasks like answering a question about a website or video summarization, you do not need to train these sub-models end to end. Instead you keep them as fixed tools and train only the main model to learn how to prompt them and use them well. Once the sub-routine is solved, you can freeze that version of the sub-model.

Q: Why is async and overlapped execution important for RL training infrastructure?

Whether you are training in a centralized or decentralized way, you ideally want to move toward everything being async and overlapped. Otherwise you get inefficiency bubbles that pop up all over your compute structure. By running everything in parallel with some lag, you can hide those bubbles. You also want infrastructure for managing the compute resources of rollouts, so that models doing tasks like data science can do them quickly as part of the training process.

Summary & Key Takeaways

• The talk sketches a roadmap toward an open-source model that behaves like o3, a general-purpose agent for long-horizon tasks such as browsing the web, writing code, and analyzing images. Although OpenAI has proven it is doable, no open-source model yet behaves this way; models like R1 excel only at single-turn math and code reasoning.

• The core path is multi-turn RL: incorporating tool calls into the loop so the model learns to use tools to solve problems, rewarded on the final output. Challenges include credit assignment across up to 100 tool calls, verifying answers outside math and code, avoiding context-length blowups, and managing rollout compute resources.

• Multimodal and multi-agent unlocks are framed as tool calls. Image manipulation like cropping and zooming refocuses attention efficiently. Rather than group-chat multi-agent systems that do not work, sub-agents act as focused tools, offloading tasks like reading websites or video summarization to smaller models to keep context chains scalable.