What Are Interactive AI Benchmarks and Why They Matter

TL;DR

Interactive benchmarks measure how efficiently an AI learns and adapts, not just what it knows. The ARC Prize Foundation's ARC-AGI-3 is a series of 150 novel open-sourced video game environments with no instructions, forcing agents to explore, plan, and act. Because intelligence is inherently interactive, static question-and-answer benchmarks cannot capture generalization or adaptation to unseen situations.

Transcript

Hi, my name is Greg Camrad, president of Arc Prize Foundation, and today we are going to learn how we measure Frontier AI. In the next 20 minutes, I'm going to step you through why interactive benchmarks are the key to doing this. We're going to take it take a look at a new Frontier AI benchmark. And then finally, we're going to wrap up with unders... Read More

Key Insights

• Intelligence is defined by Francois Chollet's 2019 paper as skill acquisition efficiency, meaning the ability to learn new things rather than mastering a single fixed task like chess, Go, or self-driving.
• The ARC Prize Foundation, started by Francois Chollet and Mike Knoop in 2024, is a nonprofit whose Northstar is open progress toward AGI, defined as a machine's ability to learn as efficiently as humans.
• Interactive benchmarks are needed because human-like intelligence shows up as agents that learn and adapt on the fly, unfolding step by step through perception, feedback, and action rather than one-shot problems.
• Static benchmarks that ask a question and get an answer cannot test exploration, the perceive-plan-act loop, memorization choices, goal and meta-goal acquisition, or an agent's alignment and cooperation capabilities.
• ARC-AGI-3 is a series of 150 open-sourced video game environments, each built on entirely new game mechanics so that no two games are alike, comparable to how Connect 4 differs from Solitaire and Pac-Man.
• The games are split into a public test set for learning the interface and a private evaluation set that neither developers nor AI have seen, so success proves genuine generalization to unseen examples.
• ARC-AGI-3 games deliberately have no natural language instructions; the test taker must click around, observe how actions change the environment, and figure out the goal through curiosity-driven exploration.
• The design philosophy targets problems that are easy for humans but hard for AI, because humans are the only proof point of general intelligence, so a game is only included if a panel of humans can solve it.

Questions & Answers

Q: What is the definition of intelligence used by the ARC Prize Foundation?

The foundation uses Francois Chollet's 2019 definition from his paper on the measure of intelligence, which defines intelligence as skill acquisition efficiency. Put another way, it is your ability to learn new things. Greg Kamradt notes that AI can already learn any single new task, such as playing chess, self-driving a car, or playing Go, but getting those same systems to learn something else still remains out of reach. The foundation also defines AGI as a machine's ability to learn as efficiently as humans.

Q: Why are static benchmarks not enough to measure AI agents?

Static benchmarks only ask a question and get an answer back, so they cannot capture interactive intelligence, which unfolds step by step through perception, feedback, and action. The real world does not give one-shot problems. Interactive benchmarks add new capabilities: testing an agent's ability to explore a new environment, run the perceive-plan-act loop, decide what to memorize when there is more information than it can hold, acquire goals and meta-goals, and demonstrate alignment and cooperation. None of these can come from a static benchmark.

Q: What is ARC-AGI-3?

ARC-AGI-3 is a new frontier AI benchmark from the ARC Prize Foundation consisting of a series of 150 open-sourced video game environments, each novel and created by the foundation itself, which built a mini game studio to make them. Each game is built on entirely new game mechanics so no two are alike, illustratively as different from each other as Connect 4 is from Solitaire and Pac-Man. The goal is to test a human or AI test taker's ability to adapt to novel situations and figure out the goals of the environment.

Q: How does the public and private test set split work in ARC-AGI-3?

The games are split across a public test set and a private evaluation set. On the public side, AI and researchers get used to the game format and interface. However, all performance metrics used to evaluate how new models are doing come from the private evaluation, which contains games neither the developer nor the AI has seen beforehand. Success on the private test set asserts that a model has genuinely generalized to unseen examples rather than just repeating what it learned on the public data.

Q: Why do ARC-AGI-3 games have no instructions?

The games intentionally provide no natural language instructions because the whole point is that the test taker must figure out what to do. As demonstrated in the live demo, a player starts by clicking around to see how their actions impact the environment, forms hypotheses about what buttons do, and confirms them through trial. Kamradt describes this as curiosity-driven exploration, which humans do naturally. Removing instructions ensures the benchmark measures genuine adaptation to a novel situation rather than following given directions.

Q: Why does ARC-AGI-3 target problems that are easy for humans but hard for AI?

The foundation loves problems that are easy for humans but hard for AI because humans are the only proof point of general intelligence. As the foundation works to make models that are more generally intelligent, humans serve as the anchor and starting point. If they can find problems that humans can solve but current AI cannot, that reveals a gap and signals something is missing. A game is only included in ARC-AGI-3 if a panel of humans can solve it.

Q: How do the ARC-AGI-3 games prevent overfitting?

Each game is designed extremely intentionally so that a test taker cannot overfit to any one game type. Kamradt explains it would be lame to make just one game type and then procedurally generate many levels of it, because that would only repeat a skill already learned once. Instead, each game is built on entirely new mechanics and is very different from every other game. This design forces genuine adaptation to novel situations rather than reapplication of a single previously mastered skill.

Q: What role does measuring efficiency play in interactive evaluations?

Interactive benchmarks measure not just intelligence but how efficient that intelligence is, tying back to the definition of intelligence as skill acquisition efficiency. In the live demo, Kamradt points out that every time a player clicks, the system records the number of actions it took to solve a particular game. This action count matters because it captures efficiency of learning and problem solving, letting evaluators compare how economically a human or agent reaches the goal in a novel environment.

Summary & Key Takeaways

• Greg Kamradt, president of the ARC Prize Foundation, explains that measuring frontier AI requires asking not whether AI is progressing but what it is progressing toward. Measuring generalization requires benchmarks built specifically to target it, starting from a clear definition of intelligence.

• Building on Francois Chollet's 2019 definition of intelligence as skill acquisition efficiency, the foundation argues human-like intelligence will arrive as interactive agents that learn and adapt on the fly. Because intelligence is inherently interactive, static one-shot benchmarks cannot measure exploration, planning, memory, or goal acquisition.

• ARC-AGI-3 delivers 150 novel open-sourced game environments with no instructions, split into public and private sets. The live demo shows games solved by clicking to test hypotheses, recording action counts, and targeting problems easy for humans but hard for AI to reveal capability gaps.