How Claude 3.5 Sonnet Builds Coding Agents

TL;DR

Strong coding agents need realistic repository tasks, carefully designed tools, and evaluation against testable software changes. Claude 3.5 Sonnet reached 49% on SWE-Bench Verified using a reference agent whose tools and prompt were released, showing how model capability and agent design combine to improve performance on real engineering work.

Transcript

hey everyone welcome to the laden space podcast this is cesio partner and CTO at deible partners and today uh we're in the new studio uh with my usual co-host Sean from small AI hey and today we are very blessed to have Eric schen from anthropic with us welcome hi thanks very much uh I'm Eric schin I'm a member of technical staff at anthropic uh wo... Read More

Key Insights

• SWE-Bench is a repository-level coding evaluation built from issues and corresponding testable changes in 12 popular open-source Python repositories. Unlike isolated programming puzzles, it requires an agent to identify relevant files and understand how a proposed modification interacts with an existing codebase.
• SWE-Bench Verified is a manually filtered version of the original benchmark. Manual review is necessary because some scraped tasks are impossible to solve from the information provided, even though a human previously completed the underlying software change with access to broader context.
• Claude 3.5 Sonnet achieved 49% on SWE-Bench Verified with Anthropic's reference agent. The result exceeded a previously cited state-of-the-art score of 45%, while the benchmark had reportedly been near 30% roughly six months earlier.
• Anthropic's product research team is focused on capabilities that customers value, rather than only abstract benchmarks such as math problems or MML. Coding agents became a priority because startups and other customers were already trying to build them with Anthropic's models.
• A reference coding agent is valuable because benchmark results depend on more than the underlying model. Anthropic released the exact tools and prompt used for its SWE-Bench work so developers could build agents on the same foundation and obtain more value from the model.
• Repository navigation is a central part of practical software engineering. SWE-Bench captures this challenge because agents rarely begin with a blank project and must instead determine where to make a change, understand existing behavior, and account for interactions across the system.
• Agentic coding evaluations are expensive and difficult to run because each task can require extensive repository parsing, repeated code edits, test execution, and many tokens. Simpler evaluations such as HumanEval therefore remain useful for quick, lower-cost signals about greenfield code generation.
• Useful AI coding marked a personal threshold for Schluntz because code-generation systems were already writing a substantial portion of his code. That experience led him to study large language models during a sabbatical and eventually join Anthropic to work closer to the technology's development.

Questions & Answers

Q: What is SWE-Bench and what does it measure?

SWE-Bench is a coding benchmark based on testable software issues from existing repositories. Its final set discussed in the interview contains 2,000 tasks drawn from 12 popular open-source Python repositories. An agent must inspect a repository, locate relevant files, understand the surrounding system, implement a change, and satisfy tests associated with the issue.

Q: How is SWE-Bench different from HumanEval?

SWE-Bench begins inside an existing software repository, while HumanEval represents a more traditional form of isolated or greenfield code generation. SWE-Bench adds the practical challenge of finding where a change belongs and understanding how it affects other systems. HumanEval remains useful because it is easier and cheaper to implement and can still provide a quick signal about coding capability.

Q: Why was SWE-Bench Verified created?

SWE-Bench Verified was created because some tasks in the broader scraped benchmark cannot be solved from the information supplied to an agent, even though a human completed the original change. OpenAI manually filtered tasks to produce the verified set. This process makes the evaluation more meaningful by reducing cases where missing context, rather than engineering ability, determines failure.

Q: How did Claude 3.5 Sonnet perform on SWE-Bench Verified?

Claude 3.5 Sonnet achieved a 49% score on SWE-Bench Verified using Anthropic's reference coding agent. The discussion identifies 45% as the prior state-of-the-art result and notes that performance had been around 30% roughly six months earlier. Anthropic also released the agent's exact tools and prompt so others could reproduce or extend its approach.

Q: Why did Anthropic build a reference coding agent?

Anthropic built a reference agent to understand how customers could obtain strong coding performance from Claude and to establish a practical implementation for SWE-Bench. Benchmark success depends on the model, its tools, its prompt, and the surrounding runtime. Publishing those components gives developers a concrete foundation for building their own coding agents rather than only reporting a score.

Q: Why are repository-level coding benchmarks expensive to run?

Repository-level evaluations consume substantial resources because an agent may need to parse much of a codebase, identify the relevant files, write a modification, run the software or its tests, inspect failures, and edit the code repeatedly. This multi-step process can use many tokens, making SWE-Bench harder and more expensive to implement than simpler isolated coding evaluations.

Q: Why did Erik Schluntz move from robotics into AI?

Schluntz became increasingly interested in AI after tools such as Copilot began writing a significant amount of his code. He viewed useful automated coding as a capability that would previously have sounded like artificial general intelligence. After leaving Cobalt Robotics, he took a sabbatical to read and research large language models, then joined Anthropic to work closer to their development.

Q: What shaped Erik Schluntz's decision to join Anthropic?

Schluntz chose Anthropic largely because he already knew people there whom he considered exceptionally smart, kind, and trustworthy. He expected the organization to be a strong cultural fit and cared about AI safety and preventing harmful uses. Anthropic's bottom-up environment also let him pursue interests in code generation, tool use, and AI systems that can affect the world.

Summary & Key Takeaways

• Erik Schluntz describes moving from robotics into AI after code-generation tools became useful in his own engineering work. At Anthropic, he joined the product research team and focused on tool use, computer use, and coding agents, with an emphasis on capabilities that customers could apply to valuable real-world tasks.

• SWE-Bench evaluates agents on software issues drawn from existing Python repositories, requiring them to locate relevant files, understand surrounding systems, modify code, and satisfy tests. SWE-Bench Verified further filters tasks manually because some original benchmark problems cannot be solved using only the information supplied to the agent.

• Anthropic built a reference agent around Claude 3.5 Sonnet and released its tools and prompt so developers could reproduce or extend the approach. The discussion also covers runtime design, indexing, sandboxing, testing, LLM-oriented tools, multi-agent systems, XML, model selection, computer use, robotics, hardware, and self-driving businesses.