What will ARC AGI 3 reveal about AI progress in 2026?

TL;DR

ARC AGI 3 highlights a persistent gap between frontier AI and human performance on abstract tasks, emphasizing that 100 percent is the human baseline and not a proof of AGI. The video also covers how private test sets, calibration of benchmarks, and new model investments (Spud, Claude) shape the near term. It notes that benchmarks can be gameable and that harnesses are not allowed for ARC AGI 3, impacting evaluation strategy.

Transcript

Two exclusive reports indicate that there will be a qualitative leap in AI performance from each of the next AI models released by OpenAI and Anthropic. For OpenAI, this has meant shutting down the Sora app to spare computing resources for the new Spud model. And for Anthropic, makers of Claude, it has meant renewed interest from the Pentagon in re... Read More

Key Insights

• ARC AGI 3 measures performance with a human-based 100 percent ceiling and uses action count as a key score metric, not speed or reflexes.
• ARC AGI 3 requires private test sets to be distinct and out of distribution compared to public data to avoid gaming the benchmark.
• Inbuilt chain of thought reasoning contributed to earlier benchmark saturation, as models learned to perform many tasks similarly from training data.
• OpenAI shelves the Sora project to conserve compute resources for the new Spud model, signaling prioritization of AGI deployment over ancillary projects.
• Anthropic faces renewed government interest, with potential revision of the Pentagon deal, suggesting policy and defense considerations weigh heavily on AI partnerships.
• ARC AGI 3 is turnbased and does not count faster actions; efficiency is penalized quadratically, shaping how models approach task solving.
• The research emphasizes that private vs public task distributions are critical to evaluating true AI capability without overfitting on demonstrations.
• Harnesses, where one model guides another, show promise in memory management but are not allowed in ARC AGI 3 challenges to keep assessments generalizable.

Questions & Answers

Q: How does ARC AGI 3 define success for an AI model, and why is 100 percent considered a limit rather than proof of AGI?

ARC AGI 3 defines success as achieving the highest possible score within its framework, with the human baseline set at 100 percent and capped there. The paper notes that no model can exceed 100 percent on this benchmark because scores are based on actions and memory rather than speed or external indicators. Consequently, reaching 100 percent indicates mastery of the tested tasks but does not constitute AGI, as the benchmark cannot measure all aspects of general intelligence beyond the defined environments.

Q: What role do private test sets play in ARC AGI 3, and why are they emphasized in the methodology?

Private test sets are described as distinct and out of distribution compared to publicly available data, which helps prevent models from gamifying the benchmark by memorizing public demonstrations. The methodology argues that relying on similar public and private data can enable shortcut strategies or overfitting, reducing the validity of comparisons across models. By keeping private sets different, ARC AGI 3 aims to more accurately reflect generalization to unseen tasks.

Q: Why is the ARC AGI 3 benchmark described as turnbased, and how does this influence scoring?

ARC AGI 3 is turnbased, meaning the model responds with actions in discrete steps rather than relying on rapid reflexes or speed. This design emphasizes planning, memory, and strategic decision making over quickness. The scoring explicitly accounts for the number of actions taken, and excessive actions can lead to discontinuation of an attempt, thereby shaping how models balance efficiency and accuracy.

Q: What does the video say about the impact of Sora and Spud on compute resources and AI development priorities?

The video notes that OpenAI reportedly shut down the Sora app to conserve computing resources for the Spud model, signaling a prioritization of AGI deployment and a single, integrated platform approach over multiple side projects. This shift reflects broader industry pressure to allocate resources toward models expected to drive faster progress or economic impact, rather than maintaining parallel experiments.

Q: How do the ARC AGI 3 findings relate to the broader debate about whether current AI approaches can achieve AGI?

ARC AGI 3 highlights a controlled gap between frontier AI and human capabilities on abstract, nonlanguage tasks. While researchers recognize significant progress, they caution that reaching 100 percent on the benchmark does not prove AGI, because the tasks are narrow, turnbased, and finite. The discussion situates ARC AGI 3 within the ongoing debate that true AGI requires broad, generalizable capacities beyond current benchmarkable performance.

Q: What is the significance of the private-public data dynamic in evaluating AI benchmarks according to the video?

The video emphasizes that public data alone can lead to overfitting or gaming of benchmarks, while private data remains hidden from model training and evaluation. A benchmark that relies on private, out-of-distribution tasks better tests generalization and resilience to adversarial or unanticipated scenarios. This distinction helps ensure fair comparisons and more reliable assessments of true AI capability.

Q: Why might there be renewed government interest in AI models like Claude, according to the video?

The video suggests that the next advances in AI may enhance both offensive and defensive cyber capabilities, prompting government interest in expanding or reviving deals with AI providers like Claude. This context indicates that policy and defense considerations can shape negotiation dynamics, access to technology, and the strategic deployment of AI capabilities beyond commercial use.

Q: What is the role of fostering abstraction in ARC AGI 3, and how does it influence the design of tests?

ARC AGI 3 explicitly foregrounds abstraction as a core element, focusing on tasks that do not rely on language or memorized knowledge. This approach stresses that models must infer goals and plan actions without explicit instructions, testing deeper cognitive functions like exploration and memory. The design aims to measure residual human-like reasoning with minimal language bias, guiding how tests are constructed and interpreted.]}

Summary & Key Takeaways

• ARC AGI 3 is framed as a test of residual gaps between frontier AI and human abilities on abstract tasks, with a strict 100 percent human baseline. It explains that current models cannot exceed human performance on the benchmark and that results depend on test distributions and action counts.

• The video discusses OpenAI and Anthropic moves, including shelving the erotica bot to free compute for Spud and renewed Pentagon interest in Claude, underscoring how corporate and government dynamics influence AI development and deployment.

• ARC AGI 3 emphasizes methodological nuances, such as the need for distinct private test sets, the restriction on harnessing, and the role of memory and planning in solving tasks, while acknowledging the broader context of AI progress in 2026.