How Does Google Train Small AI Models to Reason?

TL;DR

Supervised reinforcement learning improves small AI models by rewarding each reasoning step instead of forcing token-by-token imitation. The strongest reported sequence was SRL followed by reinforcement learning with verifiable rewards. Separately, Gemini 2.0's multi-agent AI co-scientist identified promising liver fibrosis treatments and reconstructed a bacterial gene-transfer mechanism that human researchers had spent a decade uncovering.

Transcript

This week, Google found a way to make small AIs earn their intelligence using a bizarre new training method that mixes two systems that were never meant to work together. And while that's happening, another Google team quietly built an AI scientist that cracked a decade old biological mystery. One that took human researchers 10 years to solve. And ... Read More

Key Insights

• Supervised reinforcement learning is a framework that places expert supervision inside the reward channel instead of relying only on a conventional supervised loss. It gives a small model access to correct expert trajectories while requiring the model to earn rewards through its own step-by-step decisions.
• Dense feedback is the central advantage of SRL because every generated action receives an immediate reward, even when the eventual answer is incorrect. This helps the model identify useful intermediate decisions without being forced to reproduce an expert demonstration token by token.
• The strongest mathematics training sequence reported was SRL first and RLVR second. On Qwen 2.5 7B Instruct, that combination reached 57.5 on AMC 23, 20.0 on AIME 24, and 10.0 on AIME 25.
• SRL improved code reasoning by converting 5,000 verified software engineering trajectories from Claude 3.7 Sonnet into 134,000 step-wise examples. On SWE-bench Verified, the trained model reached 14.8% in Oracle file edit mode and 8.6% end to end.
• SRL treats reasoning as action generation rather than simple next-token prediction. For each expert-trajectory prefix, the model creates a hidden reasoning section and then produces one action, which is compared with the teacher action through a lightweight string-similarity metric.
• The AI co-scientist is a Gemini 2.0 multi-agent system with generation, reflection, ranking, evolution, and meta-review roles. Humans define research goals and provide natural-language guidance, while the agents brainstorm, criticize, compare, combine, and refine scientific hypotheses.
• The liver fibrosis study validated HDAC and BRD4 inhibitors in human hepatic organoids after the AI co-scientist proposed three drug classes. Vorinostat, an FDA-approved cancer treatment, reduced fibrosis and increased the growth of healthy liver tissue in the reported organoid model.
• The bacterial study showed that the AI co-scientist could independently infer a mechanism resembling tail piracy from pre-discovery data. Its highest-ranked hypothesis proposed that CFPICs broaden their host range through capsid-tail interactions, matching the central mechanism human researchers uncovered over 10 years.

Questions & Answers

Q: What is supervised reinforcement learning for AI models?

Supervised reinforcement learning, or SRL, is a training framework that combines expert demonstrations with reinforcement-style rewards. Instead of merely training a model to copy every token in a teacher solution, SRL divides expert solutions into smaller trajectories. The model reasons privately, selects one action at each step, and receives an immediate score based on its similarity to the teacher's action.

Q: How does SRL teach small AI models to reason?

SRL teaches reasoning by reframing a solution as a sequence of decisions. For every prefix of an expert trajectory, the model generates a hidden scratchpad inside think tags and then outputs one action. That action is compared with the teacher's action using a string-similarity metric, giving the model dense feedback about each move instead of judging only the final answer.

Q: Why can supervised fine-tuning struggle on hard reasoning tasks?

Standard supervised fine-tuning can make a small model imitate long expert demonstrations token by token rather than understand the decisions behind them. In the reported tests, Qwen 2.5 7B Instruct was trained with a small dataset of about 1,000 long examples, and conventional fine-tuning caused final scores to fall below those of the base model.

Q: What training sequence produced the best mathematics results?

The researchers identified SRL followed by reinforcement learning with verifiable rewards, or RLVR, as the strongest sequence. After this combined training on Qwen 2.5 7B Instruct, the reported scores were 57.5 on AMC 23, 20.0 on AIME 24, and 10.0 on AIME 25. The authors explicitly presented SRL first, then RLVR, as the winning recipe.

Q: How well did SRL perform on software engineering tasks?

For code reasoning, researchers used Qwen 2.5 Coder 7B Instruct and 5,000 verified software engineering trajectories generated by Claude 3.7 Sonnet. These became 134,000 step-wise examples. On SWE-bench Verified, SRL reached 14.8% in Oracle file edit mode and 8.6% end to end, compared with the base model's 5.8% and 3.2%.

Q: How does Google's AI co-scientist generate scientific hypotheses?

The AI co-scientist uses several Gemini 2.0 agents with distinct scientific roles. A generation agent proposes ideas, a reflection agent identifies weaknesses, and a ranking agent compares hypotheses through an ELO-style tournament. An evolution agent merges strong ideas or explores unusual combinations, while a meta-review agent monitors the process and helps the overall system improve.

Q: What liver fibrosis treatments did the AI co-scientist identify?

The AI co-scientist proposed HDAC inhibitors, DNMT1 inhibitors, and BRD4 inhibitors as drug classes that might reverse liver fibrosis. Researchers tested the proposals in human hepatic organoids that develop fibrosis after TGF beta exposure. HDAC and BRD4 inhibitors reduced fibrosis. Vorinostat, an FDA-approved cancer treatment, also increased healthy liver tissue growth in the reported model.

Q: How did the AI co-scientist solve the tail piracy mystery?

Researchers supplied only information available before the human discovery and asked how CFPICs could spread across bacterial species. The AI generated five ranked hypotheses, with its top proposal pointing to capsid-tail interactions that enable a broad host range. This closely matched tail piracy, where CFPICs build DNA-filled heads and hijack phage tails to enter new hosts.

Summary & Key Takeaways

• Supervised reinforcement learning combines expert guidance with reinforcement-style rewards. Expert solutions are divided into trajectories, and the model generates private reasoning before choosing one action. Each action receives immediate feedback based on its similarity to the teacher's action, creating dense supervision without requiring exact imitation of the teacher's full wording.

• Tests with Qwen 2.5 7B Instruct showed that SRL improved difficult mathematics results, while SRL followed by RLVR produced the strongest reported combination. Code experiments also improved substantially after 5,000 verified software engineering trajectories were converted into 134,000 step-wise examples, demonstrating that the method extends beyond mathematical reasoning.

• Gemini 2.0's AI co-scientist coordinates specialized agents for generating, criticizing, ranking, evolving, and reviewing hypotheses. Researchers experimentally validated two proposed drug classes against liver fibrosis in human hepatic organoids. In another study, the system inferred capsid-tail interactions as the explanation for CFPIC movement across bacterial species using only pre-discovery information.