Can RL Environments Produce General Intelligence?

TL;DR

Verifiable reinforcement learning may produce increasingly capable agents, but it depends on tasks that can be repeated cheaply in deterministic, replayable environments. Many valuable real-world skills involve sparse feedback, changing conditions, and outcomes that take months or years, so general intelligence may also require sample-efficient continual learning that compresses deployment experience back into model weights.

Transcript

So here's a big research bet that all the labs are making. They think that if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we will have basically built AGI, because this kind of training will have created a kind of problem-solving agent: the kind of thing that can make progress on open-en... Read More

Key Insights

• Verifiable reinforcement learning is the major research bet described: labs hope that millions of tasks across thousands of environments will produce agents capable of pursuing open-ended goals for weeks despite errors, ambiguity, and setbacks.
• Training inefficiency may be tolerable when its one-time cost is amortized across billions of model sessions. Under this argument, deployment performance matters more than training efficiency, particularly as reinforcement learning enables agents to solve increasingly ambitious problems over longer periods.
• Long context windows could reduce the apparent need for online weight updates. If months of workplace experience fit within a session and in-context learning remains effective, an agent might acquire substantial task competence without immediately distilling every observation back into its parameters.
• Grindability is as important as verifiability for current reinforcement learning. A useful training environment must support numerous parallel rollouts from identical starting conditions inside a deterministic, replayable simulator, allowing successful and unsuccessful actions to be compared repeatedly.
• Computer use progresses slowly partly because real websites are difficult to reproduce and operate through massive parallel rollouts. Cloning applications such as Slack and Gmail could provide suitable environments, but creating high-fidelity replicas is currently described as labor-intensive and difficult to scale.
• Real-world domains resist simulation because their outcomes depend on reset-free, non-stationary environments. Building businesses, winning court cases, trading profitably, or influencing elections may require months or years before meaningful feedback appears, preventing rapid replay and controlled experimentation.
• RLVR generalization is an empirical uncertainty, not an established result. Training on containerized white-collar tasks might create broadly adaptable planning abilities, but degradation at long context suggests that short-horizon reinforcement learning may not automatically transfer to long-horizon real-world performance.
• Continual learning requires compressing useful experience into model weights rather than indefinitely expanding a context cache. Deployment reveals valuable information about organizational practices, real uses, and recurring mistakes, yet current inference compute does not productively feed those discoveries back into model improvement.

Questions & Answers

Q: What is the next proposed AI training paradigm?

The proposed paradigm trains AI systems to complete millions of verifiable tasks across thousands of diverse reinforcement learning environments. The goal is to create general problem-solving agents that can plan, recover from mistakes, handle ambiguity, and continue working on open-ended tasks for weeks. Its success depends on whether abilities learned in reproducible environments transfer to unfamiliar, long-horizon situations in the real world.

Q: Why might inefficient AI training still be economically useful?

Supporters argue that training is a one-time expense that can be amortized across billions of later sessions. From that perspective, extremely poor sample efficiency during training may matter less than the intelligence, generality, and sample efficiency displayed during deployment. Reinforcement learning has already been associated in the transcript with agents solving more ambitious problems across longer time spans, especially in coding.

Q: What makes an RL environment grindable?

A grindable reinforcement learning environment permits many parallel attempts inside a deterministic, replayable simulator. Each agent can begin from the same state, take different actions, and receive outcomes that can be compared. A software repository placed in identical containers is a strong example because a thousand agents can independently attempt the same missing feature without affecting one another or changing the shared starting conditions.

Q: Why has AI computer use progressed more slowly than coding?

Computer-use outcomes can be verifiable, but real websites are not easily farmed through thousands of identical parallel attempts. Large numbers of agents cannot repeatedly execute the same commercial checkout flow without encountering operational restrictions. Training also has less high-quality multimodal pretraining data, while building high-fidelity clones of common applications currently requires substantial labor and does not scale easily.

Q: Why is verifiability insufficient for effective reinforcement learning?

A task can have a clearly observable final result while still being unsuitable for large-scale reinforcement learning. Efficient training also requires repeatable starting conditions, many parallel rollouts, and timely feedback. Real-world processes often change while the agent acts, cannot be reset, and reveal results only after months or years, making it difficult to determine which particular actions caused success.

Q: Can RLVR training generalize to real-world expertise?

The transcript presents that possibility as an unresolved empirical question. Labs are betting that extensive training in containerized, reproducible environments will create an agent able to plan, execute, learn quickly, and acquire new skills within one session. However, possible failure to transfer from short-horizon training to long-horizon performance raises doubts about transferring from white-collar simulations to politics or business building.

Q: Why might continual learning remain necessary for AI agents?

Continual learning would let an AI preserve lessons discovered during deployment by incorporating them into its weights. Deployment exposes models to organization-specific tacit knowledge, actual user workflows, and recurring real-world mistakes that simulated training may not reveal. Without weight updates, valuable expertise developed inside one context remains temporary and can be lost when that session ends, limiting cumulative improvement.

Q: Why can context windows not fully replace weight updates?

Keeping every new observation in an ever-growing context or KV cache is described as unscalable. Human learning instead appears to compress experience into intuitions and broad knowledge, which supports abstraction and generalization. High-fidelity recall alone may even interfere with understanding abstractions and metaphors. The desired process therefore preserves important patterns in model weights rather than retaining every deployment detail verbatim.

Summary & Key Takeaways

• AI labs are betting that training agents on millions of verifiable tasks across thousands of reinforcement learning environments can create broad problem-solving ability. Supporters argue that greater scale, longer contexts, and stronger in-context learning could overcome training inefficiency and reduce the need to update model weights continually during deployment.

• Computer use illustrates why verifiability alone is insufficient. Effective reinforcement learning also requires grindable environments where many agents can start from identical conditions and perform parallel rollouts. Coding supports this structure through reproducible containers, while interactions with real websites are harder to clone, reset, replay, and operate repeatedly at scale.

• Many important capabilities must be learned through scarce, ambiguous real-world experience rather than deterministic simulations. The central uncertainty is whether reinforcement learning with verifiable rewards will generalize from reproducible tasks to politics, entrepreneurship, and other long-horizon domains. Continual learning may be needed to preserve and compress valuable deployment discoveries into model weights.