How Should You Evaluate Gemini 3.1 Pro?

TL;DR

Gemini 3.1 Pro is a frontier model whose value depends more on your domain and prompting setup than on any single leaderboard score. Its strong ARC-AGI-2, coding, and Simple Bench results coexist with weaker professional-task performance, sensitivity to question formats, and persistent hallucinations, showing why users should test models on representative work instead of declaring one universally best.

Transcript

The latest, and some would say greatest, AI model has just been released, Gemini 3.1 Pro. And in the 24 hours since release, as well as a short period of early access, I have tested it hundreds of times, and of course read its model card. But here's the thing, for the average user, I want to get beyond the headline scores, and try to give you a sen... Read More

Key Insights

• Post-training is now responsible for most LLM training compute, while internet-scale pre-training accounts for 20%. This shift lets laboratories use internal benchmarks and industry-sourced data to specialize general base models, making performance increasingly dependent on whether the post-training data and objectives match a user’s domain.
• Domain specialization is weakening the assumption that strength in one intellectual task predicts strength everywhere. Claude Sonnet 4.5 scored 12% on an Epoch AI chess puzzle benchmark, while the later Claude Opus 4.6 scored 10%, despite Opus being described as an exceptional coding model.
• Gemini 3.1 Pro is competitive with leading models across many measured domains, including coding, scientific reasoning, academic reasoning, and pattern recognition. However, its comparatively weaker head-to-head result on GDP val shows that broad professional tasks can produce a different ranking from specialized academic or coding benchmarks.
• ARC-AGI-2 performance is sensitive to how its puzzles are encoded. Gemini 3.1 Pro scored 77.1%, compared with approximately 69% for Claude Opus 4.6, but research cited in the discussion found that replacing numbers with other symbols reduced accuracy because numeric color labels can reveal unintended arithmetic patterns.
• Agentic coding works through repeated optimization toward a specified goal, which can resemble machine learning. A coding agent may produce a codebase that performs the stated task without yielding transparent internal logic, and it can overfit the specification or drift away from the user’s original concept.
• Gemini 3.1 Pro reached 79.6% on Simple Bench, placing it within the margin of error of the reported average-human baseline from nine participants. The result suggests that fair English text tests for non-specialists may no longer clearly separate the average person from frontier models.
• Question format can materially alter measured reasoning performance. Removing Simple Bench’s multiple-choice options and using open-ended answers with a blind model grader caused scores to drop by approximately 15 to 20 percentage points, while still leaving frontier models with substantial performance rather than reducing them to zero.
• Hallucination risk remains distinct from overall factual performance. Gemini 3.1 Pro led the cited AA Omniscience aggregate measure with positive 30, but 50% of its incorrect answers were hallucinations, compared with 38% for Claude Sonnet 4.6 and 34% for GLM-5.

Questions & Answers

Q: How should users evaluate Gemini 3.1 Pro?

Users should test Gemini 3.1 Pro on tasks that closely resemble their own work, including the same prompt formats, domain material, and expected outputs. Its strong results in coding, scientific reasoning, academic reasoning, ARC-AGI-2, and Simple Bench do not guarantee leadership on broad professional tasks or factual reliability. A useful evaluation should examine both successful outputs and failure behavior, especially hallucinations, specification overfitting, and sensitivity to wording.

Q: Why do AI model benchmarks increasingly contradict each other?

AI benchmarks increasingly disagree because post-training can optimize models for particular domains rather than improving every capability uniformly. Internet-scale pre-training now represents 20% of the compute spent on training LLMs, according to the transcript, while most compute goes into later refinement. If a laboratory uses internal benchmarks or industry-sourced data relevant to one field, users in that field may see results that differ sharply from broader leaderboards.

Q: What does Gemini 3.1 Pro's ARC-AGI-2 score show?

Gemini 3.1 Pro scored 77.1% on ARC-AGI-2, placing it ahead of Claude Opus 4.6 at approximately 69% and supporting the claim that the Gemini 3 series handles these puzzles cost-efficiently. However, the score does not provide a format-independent measure of intelligence. Changing numeric encodings to other symbols reduced accuracy because numeric color labels could expose unintended arithmetic patterns that helped models reach accidentally correct solutions.

Q: What is the significance of Gemini 3.1 Pro's Simple Bench result?

Gemini 3.1 Pro achieved 79.6% on Simple Bench, beating the previous record from Gemini 3 Pro and reaching within the margin of error of the reported average-human baseline among nine participants. The result marks a possible threshold for fair, English, text-only tests aimed at non-specialists. It does not mean models solve every human-easy audio, visual, or tokenization-sensitive puzzle, but it indicates substantial common-sense reasoning progress.

Q: Why did open-ended Simple Bench scores fall?

Open-ended Simple Bench scores fell because multiple-choice options can reveal clues about the intended trick or answer. An option such as zero may prompt a model to reconsider an apparently straightforward question and search for a hidden interpretation. When the options were removed, models generated answers freely and a blind grader compared them with hidden correct answers. Performance dropped by approximately 15 to 20 percentage points, showing meaningful format sensitivity.

Q: Are hallucinations solved in frontier AI models?

Hallucinations are not solved, even when a model achieves a strong aggregate factual-accuracy score. On the cited AA Omniscience measure, Gemini 3.1 Pro scored positive 30, compared with positive 11 for Claude Opus 4.6 and negative four for Claude Sonnet 4.6. Yet 50% of Gemini’s incorrect answers were hallucinations, versus 38% for Sonnet and 34% for GLM-5, revealing a separate failure dimension.

Q: Why can coding benchmark records differ from coding experience?

Coding benchmarks reward performance against defined goals, but a coding agent can optimize too narrowly for the supplied specification. The transcript compares advanced agentic coding with machine learning because agents repeatedly iterate until a target is reached, potentially producing a codebase whose internal logic is not carefully inspected. Gemini 3.1 Pro reached a record Elo on LiveCodeBench Pro, yet practical use in Cursor produced output the reviewer considered difficult to reconcile with that result.

Q: Does Gemini 3.1 Pro qualify as the best overall AI model?

The available evidence does not establish a universally best overall model. Gemini 3.1 Pro is described as highly competitive with Claude Opus 4.6 and GPT 5.3 across almost any measured domain, and it leads several coding, reasoning, and pattern-recognition tests. However, it falls behind on the cited GDP val comparison, its scores can depend on benchmark encoding and answer format, and its incorrect responses can still include hallucinations.

Summary & Key Takeaways

• Post-training now accounts for most of the compute spent training large language models, allowing laboratories to sharpen general base models for particular domains. Consequently, a model can excel at coding, scientific reasoning, or pattern recognition without leading in chess, professional tasks, factual reliability, or another area that received different optimization priorities.

• Gemini 3.1 Pro demonstrates the difficulty of interpreting benchmarks. It scored 77.1% on ARC-AGI-2 and set a Simple Bench record of 79.6%, yet encoding changes and multiple-choice clues can expose shortcuts. Open-ended Simple Bench testing still produced impressive results, although scores fell by approximately 15 to 20 percentage points.

• The practical lesson is to evaluate frontier models against representative tasks, prompts, formats, costs, and failure modes. Aggregate scores can conceal whether incorrect responses become hallucinations, whether optimization overfits a specification, or whether deeper inference is cost-effective. Gemini remains highly competitive, but its strengths do not establish universal superiority across every domain.