Claude 3 Review: Does It Beat GPT-4 and Gemini?

TL;DR

Claude 3 appears to beat GPT-4 and Gemini 1.0 Ultra across Anthropic’s published benchmarks, though the results come from Anthropic’s own website. Its coding score reaches almost 85% with zero-shot prompting versus GPT-4’s 67%. The Haiku, Sonnet, and Opus models also add vision, launch with a 200K context window, and refuse fewer requests than Claude 2.1. Read on for pricing, model differences, and hands-on test results.

Transcript

there is a brand new large language model from the company anthropic it's called claw 3 and it's really a giant leap over the previous model claw 2 and based on the top Benchmark testing that they posted they beat Gemini 1.0 Ultra the recent version that Google put out their best version and they beat gp4 the best version of chat GPT sometimes by a... Read More

Key Insights

• Claude 3 is Anthropic's newest large language model and, based on the benchmark testing posted on their website, it beats Gemini 1.0 Ultra and GPT-4, sometimes by a wide margin.
• Coding performance is Claude 3's standout benchmark, reaching almost 85% with zero-shot prompting compared to GPT-4's 67%, a massive upgrade, with some categories exceeding 95%.
• Claude 3 comes in three models named Haiku, Sonnet, and Opus, mirroring Gemini's Nano, Pro, and Ultra tiers, with Opus being the best and requiring a paid upgrade.
• All three Claude 3 models have strong vision capabilities, and in Anthropic's vision benchmarks GPT-4 did not win a single category while Claude kept pace with or beat Gemini Ultra.
• Refusal rates dropped significantly from Claude 2.1 to Claude 3, moving from closer to 25% down to around 10% or lower, reducing how often the model declines to answer.
• Claude 3 launches with a 200K context window, and Anthropic says all three models will be capable of accepting inputs of 1 million tokens or more when rolled out.
• Opus API pricing is $15 per million input tokens and $75 per million output tokens, more expensive than GPT-4 Turbo's $10 input and $30 output.
• The free version of Claude gives access to Sonnet with limited credits, while the $20-a-month Pro upgrade unlocks Opus and lets you switch between models.

Questions & Answers

Q: Does Claude 3 beat GPT-4 and Gemini 1.0 Ultra?

According to benchmark results published on Anthropic’s website, Claude 3 beats GPT-4 and Gemini 1.0 Ultra in multiple categories, sometimes by a wide margin. In coding, it scores almost 85% with zero-shot prompting compared with GPT-4’s 67%. Because Anthropic published these results, the creator treats them cautiously.

Q: What are the differences between Claude 3 Haiku, Sonnet, and Opus?

Claude 3 comes in three models: Haiku, Sonnet, and Opus. Opus is presented as the most capable model and requires Claude Pro, while Sonnet is available through the free version. A Pro subscription lets users switch between models.

Q: How much does Claude 3 Opus cost?

Claude Pro costs $20 per month and provides access to Opus. Through the API, Opus costs $15 per million input tokens and $75 per million output tokens. The free version provides limited Sonnet access, and the creator ran out of credits after about an hour of light testing.

Q: How does Claude 3 Opus API pricing compare with GPT-4 Turbo?

Opus costs $15 per million input tokens and $75 per million output tokens. GPT-4 Turbo costs $10 per million input tokens and $30 per million output tokens. The creator therefore describes Opus as expensive, while noting that its published benchmarks may justify an upgrade for some users.

Q: What context window does Claude 3 have?

Claude 3 launches with a 200K context window. Anthropic also says all three models will eventually accept inputs of 1 million tokens or more, although that capability was not available at launch. The creator describes the initial 200K window as massive.

Q: Does Claude 3 refuse fewer requests than Claude 2.1?

Anthropic’s figures show refusal rates falling from close to 25% with Claude 2.1 to around 10% or lower across the Claude 3 models. The improvement does not eliminate refusals: during one test, Opus declined to convert a screenshot into code because of copyright concerns.

Q: How capable is Claude 3 at analyzing images?

All three Claude 3 models include vision capabilities, and Anthropic’s benchmarks show Claude matching or beating Gemini Ultra in some categories while GPT-4 wins none of the listed categories. In a hands-on test, Sonnet failed to recognize a short-throw projector, but Opus correctly identified it as a sleek ultra short-throw projector.

Q: Can Claude 3 turn a screenshot into working code?

Yes, although its behavior was inconsistent. In an earlier attempt, Opus converted a screenshot of the GPT-4 Turbo API pricing page into accurate HTML, CSS, and JavaScript with working calculations. During the recording, it refused the same kind of request because of copyright concerns.

Summary & Key Takeaways

• Anthropic released Claude 3, a major upgrade over Claude 2 that, according to the top benchmark testing posted on their website, beats Google's Gemini 1.0 Ultra and OpenAI's GPT-4, sometimes by large margins, and is available right now.

• Claude 3 ships as three models, Haiku, Sonnet, and Opus, all with vision capabilities. Opus is the best and requires the $20-a-month Pro upgrade, while the free tier gives limited access to Sonnet with credits that run out quickly.

• In hands-on tests, Opus accurately identified a short-throw projector, wrote conversational copy preferred over GPT-4's promotional style, and converted a screenshot into working HTML, CSS, and JavaScript, though it sometimes refused tasks citing copyright.