How Can Agentic AI Evaluations Scale Fairly?

TL;DR

Reliable agentic AI evaluation requires open benchmarks, transparent configurations, broad domain expertise, and tests that remain useful as models improve. Kaggle is pursuing community hackathons, standardized agent exams, a competitive Game Arena, and a shared benchmark platform so more people can create, run, verify, and improve evaluations at scale.

Transcript

All right. Hi everybody. Um, let me just try to stand straight so I don't have to crouch over. Um, thank you all for coming. This is our talk on a genetic evaluations at scale for everybody. Um, I hope everyone's in the right room and if you are, thank you for coming. We were expecting like 20 people, so this is like way more than what we expected.... Read More

Key Insights

• AI benchmarks are scattered and become stale quickly because more than 10 can appear each day, while authors often move to new research after publication. Without continuing maintenance and participation, the accompanying leaderboards gradually lose their relevance for comparing current systems.
• Benchmark results are sensitive to evaluation configurations, including the harness and model-specific settings. On SWE-Bench Pro, six frontier models scored within a few percentage points of one another, yet changing the harness shifted performance by 22%, showing that infrastructure can outweigh apparent model differences.
• Published benchmark numbers are not automatically comparable because laboratories can optimize evaluation conditions for their own models. A competing laboratory reran a Kaggle benchmark using compaction provided through its API and reported much better results, although the underlying evaluation setup differed.
• Broad participation is necessary because AI researchers and technical professionals represent only a small part of human knowledge. Capabilities that are never evaluated cannot be measured or deliberately improved, allowing models to become superhuman in some areas while remaining mediocre in others.
• Domain experts can create evaluation data that AI laboratories do not possess. A Turkish wastewater treatment plant engineer used 20 years of experience to build a safety benchmark motivated by severe incidents in which failures to follow protocols resulted in deaths.
• Hackathons can direct community expertise toward defined evaluation problems while leaving room for creative solutions. Kaggle's approach provides guardrails, datasets, model access, hosting, and public writeups so globally distributed participants can contribute work that others can inspect and extend.
• Standardized agent exams reduce evaluation to a one-line prompt that an agent follows to take an exam. The resulting score can be placed on a leaderboard for comparison, making structured testing more accessible to people who do not operate research laboratories or enterprise evaluation systems.
• Competitive game evaluations resist saturation because models play directly against one another and receive ELO-style ratings. Games such as poker and chess always produce competitive outcomes, allowing the benchmark to remain open to continued improvement instead of reaching a fixed maximum score.

Questions & Answers

Q: Why are current AI benchmarks difficult to trust?

Current AI benchmarks can be difficult to trust because their model settings, orchestration methods, and evaluation harnesses are not always transparent. Small methodological differences can significantly change reported performance. On SWE-Bench Pro, changing the harness shifted performance by 22%, even though six frontier models otherwise finished within a few percentage points of one another.

Q: How can evaluation settings change an AI model's score?

Evaluation settings determine how a model receives context, manages information, uses tools, and completes benchmark tasks. A competing laboratory reran a Kaggle benchmark with compaction supplied through its own API and obtained much better results. The comparison therefore reflected both model capability and a model-specific configuration, rather than capability under identical conditions.

Q: Why do AI benchmark leaderboards become stale?

AI benchmark leaderboards become stale because more than 10 new benchmarks can appear each day, making the field difficult to track even for evaluation professionals. After publishing a paper and its initial leaderboard, authors often move to their next benchmark. Without continued model submissions, maintenance, and community attention, the original leaderboard becomes less relevant over time.

Q: Why should domain experts help create AI evaluations?

Domain experts possess practical knowledge and original data that may not exist on the web or within an AI laboratory's research priorities. A wastewater treatment plant engineer in Turkey turned 20 years of professional experience into a novel safety benchmark. His work addressed procedures connected to severe incidents in which people died after safety protocols were not followed.

Q: How can hackathons improve AI benchmark development?

Hackathons can focus the energy and expertise of many participants on clearly defined evaluation problems. Kaggle aims to provide guardrails while preserving enough freedom for creative work. Supporting participants also requires dataset hosting, access to state-of-the-art models, and understandable public writeups, especially when contributors cannot afford several model API keys themselves.

Q: What are standardized agent exams?

Standardized agent exams let a user give an agent a one-line prompt that instructs it to take an exam. Kaggle then returns a score that can be compared on a leaderboard. The experimental MVP received more than 500 submissions during its first week without promotion, indicating interest in accessible testing for consumer agents outside research laboratories and enterprises.

Q: How does the Game Arena avoid benchmark saturation?

The Game Arena keeps evaluations open-ended by making models compete directly in player-versus-player games such as poker and chess. Results feed an ELO-style rating system, and each contest produces a winner and a loser. Because progress is measured relative to competing models rather than a fixed answer set, participants can continue trying to improve their rankings.

Q: What prevents community AI evaluation from scaling easily?

Community evaluation requires more than attracting participants. Organizers must give globally distributed contributors suitable datasets, hosting, model access, and ways to explain their work through public writeups. Evaluation quality also depends on human experts because agents are not presented as reliable judges of innovation and creativity, while reaching alignment among those experts remains difficult.

Summary & Key Takeaways

• Current AI evaluations are scattered, decentralized, and quick to become stale. More than 10 benchmarks can appear each day, while published leaderboards often stop receiving attention after their associated papers are released. Model scores can also vary substantially when laboratories use different harnesses, configurations, or model-specific optimizations.

• Kaggle wants evaluation creation to include people beyond the relatively small AI research community. A wastewater treatment plant engineer in Turkey demonstrated the value of this approach by converting 20 years of professional experience into a safety benchmark based on proprietary, novel information that was not available elsewhere on the web.

• The proposed ecosystem combines benchmark-building hackathons, standardized exams for consumer agents, a Game Arena based on direct competition, and an open platform for creating and sharing evaluations. Important obstacles include supplying global participants with datasets and model access, documenting contributions clearly, and coordinating human experts who must judge creativity and innovation.