How Does RLVR Train Reasoning and Agent Models?

TL;DR

RLVR trains language models by rewarding outputs that pass objective checks, making it especially useful for mathematics, code correctness, and precise instruction following. It extends post-training beyond preference-based feedback, while agent training introduces harder problems involving tools, environmental feedback, long tasks, context compression, sparse outcomes, and reward design.

Transcript

Hey everyone, welcome to the Lidden Space podcast. This is Allesio, partner and CTO Desel, and I'm joined by Swixs, founder of Small AI. Hello. Hello. And we're excited to welcome back Nathan Lambert from AI2. Welcome. Thanks. Fun to be here. Um, I feel like I also have to say Interconnects and like the like Freedman podcast and like you and the AI... Read More

Key Insights

• Tulu 3 is an effort to compress complicated industry post-training recipes into a tractable system that researchers can modify while retaining state-of-the-art performance on a focused collection of core evaluations.
• Tulu's post-training suite contains roughly 10–15 tasks, while Lambert estimates that a frontier laboratory such as OpenAI may use hundreds of evaluations, making data creation, mixing, and evaluation management a major part of post-training.
• Preference tuning had relied heavily on the UltraFeedback dataset since models such as Zephyr beta, and Tulu's work showed how preference data could be scaled beyond that long-standing academic default.
• RLVR is reinforcement learning driven by verifiable reward signals, including mathematical answers, code correctness, and precise instruction following, rather than being limited to tasks that possess a literal ground-truth answer.
• Agent training requires an environment because each subsequent action can depend on feedback from search tools or information stores, unlike basic RLVR setups that evaluate only a model's completed string output.
• Deep research systems may work by combining reinforcement learning on smaller retrieval, editing, and search tasks with prompting and specialized fine-tunes, rather than relying entirely on reinforcement learning against the quality of a final report.
• RLHF remains relevant for outcomes that require subjective comparison, since assessing which of two generated research reports is better would require human preferences rather than a simple objective verification function.
• Long-task performance depends on skills that are difficult to verify directly, including reading files and compressing context, yet failures in these abilities can undermine extended agent workflows and increase the amount of context that must be retained.

Questions & Answers

Q: What is reinforcement learning with verifiable rewards?

Reinforcement learning with verifiable rewards, or RLVR, trains a language model using a reward function that checks whether its generated output satisfies an objective condition. Mathematical answers may be compared with ground truth, code may be tested for correctness, and precise instruction following may be checked. The name emphasizes verification because a useful task can be objectively checkable without having one unique ground-truth response.

Q: How did RLVR emerge from the Tulu 3 project?

RLVR emerged from Tulu 3's broader goal of turning complicated industry post-training practices into a tractable recipe that researchers could modify. The team was exploring reinforcement learning over model outputs and initially considered describing the method as reinforcement learning from ground truths. They adopted verifiable rewards because the concept also covers code and instruction-following tasks that can be checked despite lacking one fixed ground-truth string.

Q: How does RLVR differ from RLHF?

RLVR uses an objective function that can determine whether an output is correct or satisfies a defined condition. RLHF is needed when quality depends on human judgment, such as deciding which of two generated research reports is better. The approaches can coexist in post-training because objective tasks benefit from verification, while subjective or non-verifiable behaviors still require preferences, curated examples, or other reward signals.

Q: Why are mathematics and coding suitable for RLVR?

Mathematics and coding provide success criteria that can be checked with relatively clear functions. A mathematical response can be compared with an expected answer, while generated code can be evaluated for correctness. These domains therefore provide direct reward signals for reinforcement learning. Precise instruction following can also fit this framework when compliance with the requested constraints can be objectively verified.

Q: Why does agent training require an environment?

Agent training requires an environment because an agent's next action can depend on feedback produced by its previous action. When a model searches an information store, for example, the returned results influence what it should do next. This differs from basic RLVR, where the model produces a string and a function simply checks the completed output without an interactive sequence of tools, observations, and decisions.

Q: How might deep research systems be trained?

Lambert suggests that deep research may be built by training smaller capabilities and then combining them, rather than applying end-to-end reinforcement learning only to the final report. Relevant components include information retrieval, search, editing, and related tasks. Prompting and specialized fine-tunes can help assemble those capabilities, while judging the overall report through outcome-based reinforcement learning would resemble RLHF because report quality is subjective.

Q: What limits verifiable rewards for long agent tasks?

Many essential agent behaviors do not have simple verification functions. Reading files, extracting information softly, and compressing context can determine whether a long workflow succeeds, yet their quality may not reduce to one objective answer. Sparse rewards at the end of multiple generations also make training harder because the model must connect a distant outcome to many intermediate actions and environmental responses.

Q: How can post-training correct non-verifiable behavior?

Post-training does not have to rely exclusively on RLVR. When a model repeatedly performs a non-verifiable behavior poorly, developers can create hundreds or thousands of relevant instructions and include them in post-training. This can target information extraction, context handling, or other behaviors without a clean reward function. Reinforcement learning remains data-efficient when the right signal exists, but curated training examples provide another route.

Summary & Key Takeaways

• Nathan Lambert traces RLVR from the Tulu 3 effort to compress complicated industry post-training recipes into methods that others can modify. Tulu combined large-scale instruction tuning, preference tuning, and roughly 10–15 core evaluation tasks, with models based on Llama that matched or exceeded Meta on those selected evaluations.

• RLVR replaces subjective comparison with reward functions that can check whether an output succeeds. Mathematics can use ground-truth answers, code can be checked for correctness, and precise instruction following can be verified. The broader term, verifiable rewards, was chosen because many useful tasks are checkable without having a single ground-truth response.

• Agent training complicates the RLVR framework because later actions depend on feedback from tools and information stores. Rather than training an entire research workflow solely from a final outcome, current systems may combine reinforcement learning on smaller retrieval, search, editing, and information-extraction tasks with prompting and specialized model fine-tunes.