How GDP val measures real world AI work and progress

TL;DR

GDP val measures how models perform on economically valuable, real world tasks across multiple domains, using expert comparisons and blinded grading to track progress over time. It explains why evaluation matters for steering training and how tools are provided to replicate this process in production settings. It also clarifies limits and deployment considerations.

Transcript

Hi everyone. I'm Tel. I'm a researcher on the reinforcement learning team here at OpenAI and I lead our Frontier Evals efforts. And soon I'll be joined by Henry who leads our EVALS product. We'll be doing a Q&A at the end. So please drop your questions in the Discord and thanks so much for coming. So here at OpenAI, we're training some of the most ... Read More

Key Insights

• GDP val targets real world, economically valuable tasks rather than traditional benchmarks.
• Evaluation uses blinded, pairwise grading against expert outputs to compute a win rate.
• Tasks in GDP val are long horizon and multimodal, reflecting real work across industries.
• GDP val was released to enable the community to assess production readiness of models.
• OpenAI provides a public grader service and open source datasets for replication.
• GDP val aims to measure how AI can augment labor rather than replace it.
• The frontier evaluation plot shows progress from GPT-4 era to newer models with increasing win rates.
• Henry and Tel emphasize practical tools to evaluate and deploy AI in production environments.
• More videos from this channel:
  • How to use OpenAI o1 reasoning for coding and tasks
  • How Does Codex Improve Developer Workflows?
  • How to design frontends with OpenAI Codex multimodal
• More from this channel: OpenAI

Questions & Answers

Q: What is GDP val and why was it created

GDP val is an evaluation framework for measuring how AI models perform on real world, economically valuable tasks across industries. It was created to provide signal on model progress that goes beyond traditional academic benchmarks, enabling steering of training toward useful, real world capabilities and offering tools for the broader community to reproduce the evaluations.

Q: How does OpenAI grade models in GDP val

OpenAI uses pairwise expert grading, where a model's output is compared to a human expert deliverable in a blinded setup. Graders choose the preferred output, and these judgments are aggregated into a win rate that represents how often the model output is favored over the human deliverable, providing an unbiased measure of capability on real world work.

Q: What kind of tasks are included in GDP val

Tasks in GDP val are long horizon and multimodal, often requiring days or weeks to complete and involving tools and multiple media like images and CAD. They originate from real occupations in sectors contributing to GDP, such as real estate, manufacturing, and editing, designed to reflect practical, economically valuable work.

Q: Who created GDP val datasets

The datasets in GDP val were created by experts with an average of 14 years of experience in their fields. They designed tasks that represent the best of real world work in various occupations, ensuring that the data reflects authentic professional outputs and decision making in real tasks.

Q: How does GDP val help with production AI applications

GDP val provides a north star for product teams by offering a rigorous, reproducible way to evaluate models on real world tasks. It helps production teams estimate model readiness, compare model outputs to human performance, and decide when to deploy or iterate to improve reliability and usefulness in everyday workflows.

Q: What is the trend in model performance on GDP val

The trend shows that earlier frontier models scored low on GDP val, with modern models approaching higher win rates against experts. This trajectory indicates improving capability in real world tasks and suggests potential progress toward parity or near parity with human performance in certain complex, real world scenarios.

Q: Where can developers access GDP val tools

Developers can access GDP val tools on eval.openai.com, where they can run evaluations, and the datasets are open sourced on Hugging Face. A public grader service is also provided to help users iterate and compare results, supporting broader adoption and experimentation in the community.

Q: What limitations does GDP val acknowledge

GDP val acknowledges that evaluating real world work is complex and cannot capture every aspect of production. It clarifies that the evaluation focuses on a specified set of tasks with clear inputs and outputs, and recognises that real work involves prioritisation, iteration, feedback, and other factors beyond the scope of a single benchmark.

Summary & Key Takeaways

• GDP val is designed to cover real world work tasks across sectors that contribute to GDP, using long horizon, multimodal tasks such as real estate project briefs, CAD design, and video editing.

• OpenAI uses pairwise expert grading to compare model outputs with human experts, producing a win rate that reflects preference for model versus human deliverables and enabling trackable progress.

• The broader goal is to move beyond traditional benchmarks to measure how AI actually supports real world work while offering public tooling and a verifier service for the community.