Do CLAUDE.md and AGENT.md Files Hurt Coding?

TL;DR

Repository context files can reduce coding-agent performance when they add generic, generated, outdated, or misleading instructions. Use them selectively for essential repository-specific constraints, such as commands that must not run and required implementation patterns, while avoiding information already discoverable from files like package.json. The broader lesson is to manage context carefully because injected instructions can steer an agent away from the correct solution.

Transcript

Are you really an AI engineer if you haven't put a ton of time into your Agent MD or Claude MD files or both? Like, really? Come on, everyone's doing it. It has to be good, right? Well, what if I told you that a study just came out all about those Claude MD and Agent MD files? And uh the numbers weren't good. They're actually quite bad. Here we see... Read More

Key Insights

• The cited study found that coding agents performed worse consistently in the presented comparisons when given AGENT.md or CLAUDE.md context files. The evaluated models included Sonic 4.5, GPT 4.5, 5.1 Mini, and Qwen 3, according to the transcript.
• Repository context files are intended to tailor coding agents by supplying project-specific guidance. The study examined both generated context files on established SWE-bench tasks and developer-committed context files associated with issues from repositories that already contained such guidance.
• Context management is the central mechanism behind repository instruction files. A user's prompt enters a larger sequence containing higher-priority instructions and conversation history, so the agent produces its answer based on all available text rather than on the latest request alone.
• Instruction priority is hierarchical in the speaker's explanation. Provider instructions sit above the system prompt, the developer message occupies another layer, and user prompts have lower priority, meaning repository guidance delivered as developer context can strongly affect how an agent interprets a task.
• Generic repository documentation can waste context without giving the agent meaningful assistance. The example AGENT.md repeated ordinary package scripts such as development, linting, checking, testing, watch mode, and generation commands that were already available from package.json.
• Repository-specific operational warnings can still provide useful guidance. The example file instructed the agent not to run the development command because it should assume the server was already running, and it described the build command as intended only for continuous integration.
• Misleading technology references can steer agents toward inappropriate implementations. The speaker observed that mentioning TRPC in the repository context confused the model and caused it to try using TRPC in places where it should not be used.
• Persistent coding rules can silently shape a codebase beyond an individual user's preferences. A requirement to include return validators on Convex functions explained why the repository repeatedly used them, even though the speaker did not personally favor always including those validators.

Questions & Answers

Q: Do CLAUDE.md and AGENT.md files improve coding agents?

The presented evidence does not show a general improvement. In the cited study's comparisons, coding agents performed worse consistently when given AGENT.md or CLAUDE.md files. The study examined generated files on established SWE-bench tasks and developer-committed files from repositories. The result suggests that adding persistent repository instructions is not automatically beneficial and can steer an agent away from successful task completion.

Q: Why can repository context files make coding agents worse?

Repository context files can make agents worse when they contain generic, redundant, generated, outdated, or misleading guidance. Because the file becomes part of the agent's context, its instructions influence later decisions. The example file mentioned TRPC, which confused the model and encouraged inappropriate use. It also repeated basic scripts already available in package.json, consuming attention without necessarily helping solve the requested task.

Q: What information belongs in an AGENT.md or CLAUDE.md file?

The transcript supports including concise repository-specific constraints that an agent would not safely infer on its own. Examples include instructions not to run the development command because a server is assumed to be active, a warning that the build command is only for continuous integration, important architecture locations, and genuinely required implementation patterns. Generic script listings and misleading service references should be removed or carefully audited.

Q: How do context files influence an AI coding agent?

Context files influence an agent by adding instructions before or alongside the user's immediate request. The model generates its response from the complete available context, which can include provider instructions, a system prompt, a developer message, prior user messages, prior outputs, and repository guidance. As a result, persistent instructions can affect tool choices, architecture decisions, coding style, and the interpretation of later prompts.

Q: What is the instruction hierarchy described for AI agents?

The transcript describes a hierarchy beginning with provider instructions, followed by the system prompt, then a developer message, and finally user-level prompts. Higher-priority instructions take precedence over lower-priority ones. The Pig Latin example demonstrates this behavior: a system prompt requiring Pig Latin continued to control the answer even after the user asked the model to stop responding that way.

Q: Should package scripts be copied into repository context files?

Copying ordinary package scripts into a context file may add little value when the same information is already visible in package.json. The example file listed development, lint, lint-fix, check, test, test-watch, and generation commands, which the speaker characterized as basic information. More useful entries explain non-obvious constraints, such as why a command must not run or when a build command is appropriate.

Q: How should developers audit existing CLAUDE.md files?

Developers should inspect each instruction for accuracy, necessity, and unintended steering. The transcript provides two concrete audit findings: mentioning TRPC caused the model to use it where it did not belong, and a rule requiring return validators on Convex functions shaped code in a way the speaker did not personally prefer. Generated explanations and information duplicated from package.json are also candidates for removal.

Q: What broader lesson does the study offer about AI coding?

The broader lesson is that more context is not necessarily better context. Repository files, skills, system instructions, developer messages, and conversation history all compete to shape an agent's next output. Persistent guidance should therefore be treated as an active behavioral control, not harmless documentation. Developers should keep only accurate, task-relevant constraints and evaluate whether those instructions improve real task completion.

Summary & Key Takeaways

• A cited study evaluates whether repository context files improve coding-agent task completion. It compares generated files on established SWE-bench tasks with developer-committed files from real repositories. Across the presented comparisons involving several model families, agents consistently performed worse when supplied with an AGENT.md or CLAUDE.md file, challenging a widely encouraged development practice.

• Repository files can contain useful operational constraints, architecture notes, service mappings, coding patterns, and framework guidelines. They can also duplicate package.json, preserve generated filler, or include instructions that cause unwanted behavior. In the presented repository, mentioning TRPC confused the model, while a return-validator rule produced code the speaker did not personally prefer.

• Context files influence agents because user requests are only one part of the model's input. Provider instructions, system prompts, developer messages, conversation history, and repository guidance all shape subsequent output. Higher-priority instructions can override lower-priority requests, so unnecessary or inaccurate repository guidance may steer an agent even when the immediate task suggests another approach.