How to Improve Claude Skills with Autoresearch

TL;DR

Improve Claude Code skills with Autoresearch by repeatedly measuring an evaluation pass rate and refining the skill instructions against that objective metric. The workflow needs three ingredients: a measurable metric, a reliable automated measurement tool or test suite, and a prompt or markdown file that agents can change. Read on for the exact roles of evals, repeated tests, and the Autoresearch files that matter.

Transcript

I freaking love Cloud Code skills. I think you do, too. But sometimes they're a little bit unreliable. I would say about 70% of the time I run a skill, I get an intended output. About 30% of the time, it's a bag of rocks. What I wanted to do in this video is I wanted to show you how to combine Claude Code skills with a new development in the AI spa... Read More

Key Insights

• Autoresearch is a method developed by Andrej Karpathy to autonomously optimize processes.
• Claude Code skills can be improved by combining them with autoresearch, enhancing their reliability.
• Objective metrics are essential for measuring the success of skill improvements.
• A measurement tool, ideally automated, is necessary to evaluate skill performance.
• Skill instructions or prompts are continuously refined to achieve better results.
• Running skills multiple times helps in evaluating and improving their output quality.
• Binary yes/no questions are effective for evaluating the quality of skills.
• Autoresearch can be applied beyond skills, such as websites, emails, and landing pages.

Questions & Answers

Q: How do you improve Claude Code skills with Autoresearch?

Define an objective metric such as the skill’s evaluation pass rate, create a test suite that measures outputs against your criteria, and let an agent revise the skill instructions. Repeat the tests and revisions so the prompt becomes more reliable and airtight over time.

Q: What is Autoresearch?

Autoresearch is a method presented in Andrej Karpathy’s GitHub repository for using a team of agents to autonomously optimize a process. The original process was training nanoGPT, while this workflow applies the same approach to improving a Claude Code skill.

Q: What three ingredients does Autoresearch require?

It requires an objective metric, a measurement tool, and something that can be changed. For Claude Code skills, those correspond to an evaluation pass rate, an agent-written test suite, and the skill instructions or markdown prompt.

Q: How should Claude Code skill performance be measured?

Measure it with an evaluation pass rate rather than subjective impressions such as whether an output feels better. The evaluation criteria should be run through a test suite that executes the first evaluation, then the second, then the third, and so on.

Q: Why should the measurement process be automated?

The measurement tool should ideally be automated and reliable, with no human in the loop. This lets agents consistently test proposed skill changes and use the resulting evaluation pass rate to decide whether the instructions improved.

Q: Which files from the Autoresearch repository matter for improving skills?

The repository has three highlighted files: prepare.py, train.py, and program.md. For skill optimization, prepare.py can be ignored because it covers machine-learning-specific work, while train.py can be treated like skill.md and program.md supplies instructions to the agent.

Q: What part of a Claude Code skill does Autoresearch change?

It changes the skill instructions, the prompt stored in the markdown file. An agent modifies that prompt based on evaluation results, aiming to make it better and more airtight over successive iterations.

Q: Can Autoresearch optimize processes other than Claude Code skills?

Yes; the transcript also describes applying it to a website and cold email campaigns. In the website example, about 67 tests reduced load time from roughly 1,100 milliseconds to 67 milliseconds, while the cold email example uses reply rate as its metric.

Summary & Key Takeaways

• Autoresearch, a method by Andrej Karpathy, can autonomously optimize Claude Code skills, enhancing their reliability and accuracy. By setting objective metrics and using automated measurement tools, users can iteratively refine skill prompts, leading to continuous improvement. This approach allows skills to self-improve over time, making them more effective and efficient.

• The process involves setting up an evaluation test suite and running skills multiple times to measure performance against predefined criteria. By using binary yes/no questions, users can effectively assess skill quality and make necessary adjustments to prompts. Autoresearch can be applied to various domains beyond skills, offering broad utility.

• The video demonstrates how to implement autoresearch on a diagram generator skill, showcasing its potential to optimize skill performance. By automating the evaluation and improvement process, users can achieve significant enhancements in skill accuracy and efficiency, making autoresearch a valuable tool for AI optimization.