How to Progress Through Five Levels of AI Coding

TL;DR

AI coding becomes effective when developers shift from reviewing generated code to defining precise specifications and evaluating observable outcomes. The five-level framework progresses from autocomplete to autonomous software factories, but reaching the highest levels requires external behavioral scenarios, simulated services, strong judgment, and the willingness to treat implementation as a black box.

Transcript

90% of cloud code was written by claude code. Codeex is releasing features entirely written by codecs. And yet most developers using AI empirically get slower, at least at first. The gap between these two facts is where the future of software lives. Imagine hearing this at work. Code must not be written by humans. Code must not be even reviewed by ... Read More

Key Insights

• AI coding autonomy is divided into five levels that progressively move humans away from writing code and toward directing systems, specifying behavior, and evaluating outcomes. The framework distinguishes simple code completion from a dark factory that converts specifications into working software without human coding or review.
• Level zero is spicy autocomplete, where the developer writes the software and the AI suggests subsequent code. The human accepts or rejects each suggestion, so the tool primarily reduces typing effort rather than assuming responsibility for architecture, implementation, or integration.
• Level one is the coding intern model, where AI completes discrete, well-scoped tasks such as writing a function, building a component, or refactoring a module. The human still reviews everything and remains responsible for architecture, judgment, integration, and the quality of the completed system.
• Level two is the junior developer model, where AI navigates a codebase, understands dependencies, and performs changes spanning multiple files or modules. Humans continue reading the generated code, even though the scope and complexity of the AI's implementation work have expanded significantly.
• Level three is the developer-as-manager model, where AI performs implementation and submits feature-level or pull-request-level work. Humans primarily direct, approve, or reject results, but they still need enough technical judgment to evaluate whether the proposed changes should be accepted.
• Level four is the developer-as-product-manager model, where humans write specifications and later evaluate whether tests pass. Code becomes a black box, so success depends on comprehensive specifications and evaluations that establish confidence through observable behavior rather than line-by-line inspection.
• Level five is the dark factory model, where specifications enter an autonomous system and working software emerges without human-written or human-reviewed code. StrongDM's three-person team is presented as a production example built around AI agents orchestrated through markdown specification files.
• External scenarios are behavioral evaluations stored outside the codebase and hidden from the development agent. They act as a holdout set, preventing the agent from optimizing directly for visible tests and helping determine whether the resulting software works from an external user's perspective.

Questions & Answers

Q: What are the five levels of AI coding?

The framework progresses from limited assistance to autonomous production. Level zero is autocomplete, level one delegates discrete tasks, and level two assigns multifile feature work. At level three, the human manages AI-produced implementations. At level four, the human writes specifications and evaluates outcomes. Level five is a dark factory where no human writes or reviews the code.

Q: Why can AI coding tools make developers slower?

The cited 2025 randomized control trial by METR found that experienced open-source developers using AI tools took 19% longer to complete tasks than developers working without them. Those developers nevertheless believed AI made them 24% faster. The result suggests that added prompting, review, correction, and integration work can outweigh typing savings, while subjective impressions can misrepresent actual productivity.

Q: What happens at level three of AI coding?

At level three, the developer acts as a manager rather than the primary implementer. The AI builds features and submits work at the pull-request level, while the human directs tasks and decides whether to approve or reject the output. Technical judgment remains essential because the human still evaluates implementations, even though the model performs the coding itself.

Q: How is level four AI coding different from level three?

Level three still centers on human review of feature-level or pull-request-level implementations. At level four, the human behaves more like a product manager by writing a specification, leaving the agent to work, and later checking whether evaluations pass. The code is treated as a black box, making specification quality and outcome-based validation more important than direct code inspection.

Q: What is a dark software factory?

A dark software factory is a level-five system that converts specifications into working software without humans writing or reviewing the code. AI agents interpret specifications, implement the system, test its behavior, and ship the result independently. Humans concentrate on describing requirements and evaluating outcomes, while machines perform every implementation step between those activities.

Q: Why does StrongDM use external scenarios instead of traditional tests?

Traditional tests live inside the codebase, so an AI agent can read them and optimize specifically for passing them without necessarily producing correct behavior. StrongDM's scenarios are external behavioral specifications stored separately from the development environment. Because the agent cannot see the evaluation criteria, the scenarios function as a holdout set and reduce opportunities to game the assessment.

Q: What is a digital twin universe for AI development?

A digital twin universe is a simulated environment containing behavioral clones of external services used by the software. StrongDM's examples include simulated versions of Okta, Jira, Slack, Google Docs, Google Drive, and Google Sheets. Agents can run full integration scenarios against these twins without accessing real production systems, APIs, or data.

Q: How can developers progress beyond level two AI coding?

Progress requires changing the developer's role from code author to system director. Developers must learn to define complete specifications, establish behavioral evaluations, judge feature-level outcomes, and trust validated results without reading every generated line. The StrongDM example also indicates that hidden scenarios and simulated external services are important infrastructure for safely delegating longer and more autonomous development workflows.

Summary & Key Takeaways

• The five levels describe increasing AI autonomy: autocomplete assistance, isolated task completion, multifile feature work, feature-level management, outcome-based product management, and fully autonomous software production. At each level, humans move farther away from implementation and focus more heavily on architecture, specifications, judgment, evaluation, and acceptance of results.

• Most developers remain between levels one and three, where humans still scope tasks, inspect generated code, and manage integration. The cited METR trial found that experienced open-source developers using AI took 19% longer, despite believing they were 24% faster, showing that perceived productivity can differ sharply from measured performance.

• StrongDM's three-person software factory represents the level-five model described in the video. Markdown specifications direct AI agents that implement and evaluate software. Hidden external scenarios reduce test gaming, while digital twins of services such as Jira, Slack, and Google Drive support integration testing without touching production systems, APIs, or data.