How Good Are the New Claude 3.5 Sonnet and Haiku Models?

TL;DR

The upgraded Claude 3.5 Sonnet performed well on most reasoning tests but remained unreliable for exact word counting and checkers programming. It correctly solved the 25-horse puzzle in seven races, yet reported 88 words where Microsoft Word and a manual count found 107. Claude 3.5 Haiku and the experimental computer-use API broaden the release beyond reasoning alone. Read on for the test results, limitations, and autonomous workflow demonstration.

Transcript

Claude just introduced two new AI large language models one of them is Claude 3.5 sonnet but this is a brand new version of it with much better reasoning and coding abilities and they also introduced a brand new claw 3.5 ha cou so far we had claw 3 Hau so this is a lighter weight version of it and they also introduced a brand new way to use AI call... Read More

Key Insights

• Release scope matters: The announcement covers more than one upgraded language model. It combines the new Claude 3.5 Sonnet, the lighter Claude 3.5 Haiku, and an experimental computer-use capability. That combination positions the release around reasoning, coding, lower-cost user-facing applications, and direct interaction with graphical computer interfaces.
• Availability differs by product: The upgraded Claude 3.5 Sonnet was available during the review and could be selected for immediate testing. Claude 3.5 Haiku was not yet available to the reviewer and was expected within the next few weeks. Computer use was already offered through the API as an experimental public beta for developers.
• Adoption trails stated preference: The analytics shown in the review report 70 million monthly visits for Claude’s website and 3.1 billion for ChatGPT’s website. The reviewer considers this gap surprising because he uses Claude more often and regarded the earlier Claude 3.5 Sonnet as the leading model for his everyday tasks.
• Benchmark context needs caution: The displayed comparison places the upgraded Sonnet ahead of GPT-4o and Gemini 1.5 Pro, but the reviewer notes that OpenAI’s o1 reasoning model is missing. He considers that omission relevant because Sonnet is being presented as an improved reasoning model, while GPT-4o is not OpenAI’s reasoning model in his comparison.
• Computer control is visual: Computer use operates by taking screenshots and interpreting what appears on the screen. Claude can then move the cursor, click controls, type text, and navigate between interfaces. The demonstration therefore shows interaction with existing tools through their visible interfaces, rather than merely generating instructions for a person to follow.
• The demo crosses applications: Claude begins with a spreadsheet, discovers that Ant Equipment Company is absent, and switches to a CRM. It searches for the company, scrolls through the matching record, collects the necessary information, and enters it into a vendor form. The value shown is the completion of a connected, multi-interface workflow.
• Routine work drives the example: Anthropic’s fictional vendor-request task represents administrative work in which required information is scattered across different places. Claude handles the lookup, transfer, form completion, and submission steps autonomously. The reviewer sees this as a strong example of the promised direction for AI agents, while acknowledging that the capability remains an early experimental beta.
• Simple comparisons succeed: Sonnet correctly determines that 9.9 is larger than 9.11, avoiding a common mistake caused by treating decimals like whole-number strings. It also finds three Rs in strawberry. These compact tests show successful handling of basic comparison and letter-counting prompts, although they do not establish reliability on unrelated tasks.
• Familiar riddles are handled: The model identifies M as the letter appearing in minute and moment but not in a thousand years. It also explains that two fathers and two sons can be only three people when they are a grandfather, father, and son. Both answers reflect successful interpretation of wording rather than lengthy computation.
• Constraint reasoning reaches seven: In the 25-horse puzzle, five horses can race at once, no stopwatch is available, and the goal is to identify the three fastest horses. Claude concludes that seven races are necessary. The result demonstrates stronger multi-step constraint reasoning than the short letter and decimal questions used earlier.
• Visible arithmetic can mislead: Claude presents a word-by-word process and concludes that the tested response contains 88 words. Microsoft Word reports 107, and the reviewer independently counts 107 manually. The disagreement shows that exposing intermediate counting steps does not guarantee an accurate total, making external verification necessary for this exact task.
• Interface polish masks defects: Claude’s revised checkers program looks visually good, but appearance is not matched by correct gameplay. The initial version uses an unwanted terminal format, the next standalone version has broken logic, and the third attempt still cannot perform captures properly. Three prompts therefore fail to produce a fully functional checkers game.

Questions & Answers

Q: How does Claude 3.5 Sonnet perform in reasoning and coding tests?

Claude 3.5 Sonnet passes most of the demonstrated reasoning tests but performs unevenly on precision and coding. It correctly counts three Rs in strawberry, compares 9.9 with 9.11, solves two familiar riddles, and determines that the 25-horse puzzle needs seven races. It fails the exact word-count test by reporting 88 instead of the verified 107. Its Python checkers project also remains functionally broken after three prompts, showing that strong reasoning results do not guarantee dependable counting or executable game logic.

Q: What improvements does the new Claude 3.5 Sonnet offer?

The new Claude 3.5 Sonnet is presented as an upgrade with much better reasoning and coding abilities. The reviewer already preferred its earlier version for summarization, article writing, marketing, and other everyday tasks. Direct testing supports the reasoning claim across several short puzzles and the more demanding 25-horse problem. The coding claim receives weaker support because the checkers application still has movement and capture problems after repeated revisions.

Q: Can Claude 3.5 Sonnet count words accurately?

Claude 3.5 Sonnet does not count the tested passage accurately. It displays a word-by-word counting procedure and returns a total of 88. Microsoft Word counts the same response as 107 words, and the reviewer’s manual count also produces 107. The test matters because the detailed-looking process could appear trustworthy even though its final total is wrong.

Q: Can Claude 3.5 Sonnet build a working checkers game?

Claude 3.5 Sonnet does not produce a fully working checkers game within the three prompts shown. Its first Python output runs as an unwanted terminal-style application instead of the requested standalone format. The next version has an attractive interface, but its gameplay logic is broken. A third request improves initial movement, yet captures still fail, so visual quality and partial progress do not amount to a functional game.

Q: What is Claude computer use?

Computer use is an experimental public beta available through the API and intended primarily for developers. It lets Claude examine screenshots, move a cursor, click buttons, type text, and navigate computer interfaces. This enables Claude to act across visible applications instead of only returning text to the user. The demonstration uses these abilities to complete an entire vendor-request workflow with no ongoing manual execution of each step.

Q: How does the computer-use vendor demo work?

Claude first inspects a spreadsheet to see whether Ant Equipment Company is present. After determining that it is missing, Claude moves to a CRM, searches for the company, and scrolls through its record to gather the required information. It then transfers those details into a vendor request form and submits the completed form. The workflow demonstrates why computer use could reduce drudge work involving searches, data transfer, and form completion across multiple interfaces.

Q: What is Claude 3.5 Haiku designed for?

Claude 3.5 Haiku is described as a lighter model combining affordability with strong performance. It is positioned for fast, user-facing products where response speed and cost are important. The model follows Claude 3 Haiku but was not available for direct testing during this review. It was expected to roll out within the following weeks, so the demonstrated test results concern the upgraded Claude 3.5 Sonnet instead.

Q: How do Claude and ChatGPT compare in popularity and benchmarks?

The website analytics shown report 70 million monthly visits for Claude and 3.1 billion for ChatGPT, giving ChatGPT a much larger audience in this comparison. The reviewer finds that gap surprising because he uses Claude more and considers Sonnet stronger for his everyday work. The displayed benchmarks place the upgraded Sonnet ahead of GPT-4o and Gemini 1.5 Pro. He nevertheless warns that OpenAI’s o1 reasoning model is absent, which limits what the benchmark comparison can establish about competing reasoning systems.

Summary & Key Takeaways

• Introducing the upgraded models: Claude 3.5 Sonnet arrives with claimed improvements in reasoning and coding, alongside the lighter Claude 3.5 Haiku. The reviewer already regarded the earlier Sonnet as the best available model and preferred Claude for summarization, article writing, marketing, and other everyday work. Sonnet was available to select immediately, while Haiku was expected to roll out over the following weeks as an affordable, high-performance option for fast, user-facing products.

• Comparing adoption and benchmarks: The review contrasts Claude’s reported 70 million monthly website visits with ChatGPT’s 3.1 billion, despite the reviewer’s preference for Claude. Presented benchmarks show the upgraded Sonnet ahead of GPT-4o and Gemini 1.5 Pro. The reviewer adds an important qualification: OpenAI’s o1 reasoning model was absent from that comparison. Because published benchmarks cannot establish practical reliability by themselves, the review proceeds to direct reasoning and coding tests.

• Demonstrating autonomous computer use: Anthropic presents computer use as an experimental public beta available through the API and intended primarily for developers. Claude can inspect screenshots, move a cursor, click buttons, type, and navigate interfaces. In the fictional demonstration, it checks a spreadsheet for Ant Equipment Company, searches a CRM when the company is missing, gathers the required details, transfers them into a vendor request form, and submits that form without continued human action.

• Testing reasoning and precision: Sonnet correctly counts three instances of R in strawberry, determines that 9.9 is greater than 9.11, and answers the minute-and-moment letter riddle with M. It also explains how two fathers and two sons can represent three people. On the harder 25-horse constraint puzzle, it reaches the correct result of seven races. Exact counting proves weaker, however, because its answer of 88 words conflicts with both Microsoft Word and the reviewer’s manual total of 107.

• Building and revising checkers: The coding trial asks Claude to create a Python checkers game, but its first result uses an unwanted terminal-style format. A follow-up produces a visually appealing standalone interface, although the game logic does not work correctly. After a third prompt, starting movement improves, but pieces still cannot capture opponents properly. The sequence shows that polished presentation and iterative revisions do not necessarily yield functional code, even when the model is introduced as having stronger coding abilities.