How to Automate Business Improvements With AI

TL;DR

Use an autonomous AI loop that repeatedly changes an editable asset, measures the result with a protected objective score, keeps improvements, and discards failures. The approach works best when outcomes can be scored objectively, feedback arrives within minutes or hours, and the AI can directly modify the asset, making website speed, email opens, and click-through rates stronger candidates than slow-moving metrics such as SEO rankings or churn.

Transcript

Andre Kapathy, the godfather of modern AI, has just released a simple system called Auto Research, which the top AI labs in the world have been spending millions trying to create. Since releasing this auto researched, it's got over 85,000 stars on GitHub. And even Shopify CEO pointed at Shopify's code and sped it up by 53%. But here's the thing, it... Read More

Key Insights

• Auto Research is an autonomous experimentation loop that changes an asset, measures the result, keeps winning variations, and rejects losing ones. The agent can repeat this process indefinitely until it reaches a specified goal or a human stops it.
• The system uses three distinct components: human-controlled instructions, an asset the AI may modify, and a protected scoring mechanism. Keeping the instructions and scoring file inaccessible to the agent prevents it from changing the rules merely to produce a higher score.
• Objective measurement is essential because the agent needs an unambiguous basis for comparing experiments. Website load speed, content impressions, email opens, and page click-through rate provide measurable outcomes, while requests to make something better or funnier do not supply a sufficient score by themselves.
• Fast feedback enables more experiments and more opportunities for measurable improvement. Results that arrive within minutes or hours are suitable, while SEO changes that may require 10 days for reindexing and pricing changes tied to churn after 6 months create feedback loops that are too slow.
• Direct access to the target asset is mandatory because the agent must be able to implement each proposed variation. HTML files and assets exposed through software APIs are workable, while the introduction of an already published YouTube video cannot be changed after publication.
• The optimization process resembles natural selection because each new variation competes against the current baseline. A superior result replaces the existing asset and becomes the basis for the next experiment, while an inferior result is discarded before another variation is attempted.
• Karpathy's reported experiment ran in 5-minute loops for 83 experiments and produced an 11% improvement. The example illustrates how an autonomous agent can continue testing changes overnight, even after a human believes that the existing system has already been extensively optimized.
• Shopify CEO Tobi Lütke reported a 19% score improvement after 8 hours and 37 experiments, then later reported that a Liquid codebase became 53% faster with 61% fewer object allocations. He also cautioned that the result was probably somewhat overfit.

Questions & Answers

Q: What is the Auto Research system?

Auto Research is a method for letting an AI agent improve a chosen asset through repeated experiments. A human specifies the objective and operating instructions, then the agent modifies the asset, measures the new version against a fixed scoring mechanism, keeps changes that improve the score, and discards changes that do not. The loop can continue overnight without ongoing human intervention.

Q: How does the three-file Auto Research setup work?

The setup separates the workflow into instructions, an editable asset, and a scoring mechanism. The human controls the instructions that define what should improve. The AI receives access to the target asset so it can create variations. The scoring file evaluates those variations but remains locked, preventing the agent from manipulating the measurement rather than genuinely improving the asset.

Q: Why must the scoring mechanism be protected from the AI?

The scoring mechanism must remain protected so the agent cannot obtain a better result by rewriting the evaluation rules. If the AI can change both the asset and the test, a higher score may not represent real improvement. Locking the score forces every variation to compete under the same measurement and makes comparisons with the current baseline more meaningful.

Q: What happens when an Auto Research experiment succeeds or fails?

When a variation achieves a better objective score than the current asset, the system keeps that variation and uses it as the new baseline for the next experiment. When a variation performs worse, the system reverts to the previous version and tries a different change. This repeated selection process resembles natural selection applied to machine-generated variations.

Q: What requirements make a business task suitable for Auto Research?

A suitable task must meet three requirements described in the transcript. Its outcome must be objectively measurable, the feedback should arrive within minutes or at most hours, and the AI must have access to modify the target asset. Website speed, email opens, impressions, and click-through rate fit these conditions more closely than subjective judgments or delayed business outcomes.

Q: Why is fast feedback important for autonomous AI optimization?

Fast feedback allows the agent to evaluate a change promptly and begin another experiment, increasing the number of useful iterations it can complete. Website load speed can be tested in seconds, and email opens can be measured within an hour. SEO rankings may require 10 days for reindexing, while pricing effects on churn may take 6 months, making both difficult targets for this loop.

Q: Can Auto Research improve assets outside software code?

The method is presented as a general recipe that can be applied beyond code, including emails, ads, landing pages, AI skills, AI agents, and organic content. The decisive issue is not the asset's category but whether the agent can edit it, receive rapid feedback, and compare variations using a reliable objective score without changing that scoring method.

Q: What results were reported from overnight Auto Research experiments?

Karpathy's example reportedly completed 83 experiments in 5-minute loops and found an additional 11% improvement after he had already worked extensively on the agent. Shopify CEO Tobi Lütke reported a 19% score increase after 8 hours and 37 experiments. He later reported 53% faster Liquid code with 61% fewer object allocations, while noting possible overfitting.

Summary & Key Takeaways

• Auto Research is presented as a reusable method for autonomous experimentation rather than a tool limited to machine-learning code. A human defines the goal and instructions, while an AI agent repeatedly changes a selected asset, evaluates each variation, retains measurable improvements, and continues working without ongoing human involvement, including through the night.

• The system separates responsibilities across three files: protected instructions, an editable asset, and a protected scoring mechanism. Restricting the agent from changing its instructions or evaluation method helps preserve an honest comparison. The agent can modify only the target asset, then keep successful versions or revert unsuccessful experiments before beginning another iteration.

• A suitable optimization target must support objective scoring, rapid feedback, and direct AI modification. Website load speed, impressions, email opens, and page click-through rates fit these requirements better than subjective quality, delayed SEO rankings, long-term churn, or elements of previously published videos that the agent cannot edit after publication.