How Does AI Affect Developer Productivity?

TL;DR

AI coding tools produce an average net productivity gain of roughly 15–20% across the studied organizations, after accounting for the extra rework they create. Results vary substantially by task complexity, codebase maturity, language popularity, and codebase size, so organizations should measure delivered functionality and recent-code rework instead of assuming that more commits or pull requests represent greater productivity.

Transcript

in January of this year Mark Zuckerberg said that he was going to replace all of the mid-level engineers at Meta with AI by the end of the year i think Mark was a bit optimistic and he was probably acting like a good CEO would to inspire a vision and also probably to keep the Facebook stock price up but what Mark also did was create a lot of troubl... Read More

Key Insights

• AI coding produces an average net productivity gain of roughly 15–20% across the studied industries and sectors. Gross output may increase by approximately 30–40%, but part of that apparent improvement is consumed by fixing bugs and cleaning up code introduced during AI-assisted development.
• Productivity cannot be measured reliably by counting commits, pull requests, or completed tasks. Task sizes differ, and AI can create additional bug-fixing tasks, so rising activity may represent developers spinning their wheels rather than delivering proportionally more useful functionality.
• Greenfield experiments favor AI because coding tools perform particularly well on boilerplate work created from scratch. Most professional software engineering occurs within existing codebases with dependencies and contextual constraints, which makes results from isolated greenfield experiments difficult to generalize to company settings.
• Developer self-assessments are weak predictors of measured productivity. In an experiment involving 43 developers, participants misjudged their productivity by about 30 percentile points, and only one in three estimated their performance within the correct quartile.
• The study measures productivity by analyzing the functionality delivered in each source-code change. Because every commit has an author, timestamp, and unique SHA, researchers can aggregate evaluated changes across time to examine the output of individuals, teams, and organizations.
• Rework is distinct from refactoring because it changes code written much more recently. A rise in rework after AI adoption indicates that some additional coding activity is wasteful, even when developers feel more productive because they are writing and pushing a larger volume of code.
• AI performs better on lower-complexity tasks than on higher-complexity tasks in the enterprise data. Low-complexity greenfield work shows a higher and more extended distribution of productivity gains, while complex work and brownfield environments produce less favorable results.
• AI adoption is not a one-size-fits-all productivity strategy. Outcomes vary according to task complexity, codebase maturity, language popularity, and codebase size, and some teams can experience productivity declines rather than gains after incorporating AI into their workflow.

Questions & Answers

Q: How much does AI improve developer productivity?

AI-assisted coding increases gross coding productivity by roughly 30–40% in the study's overall interpretation, but that figure does not represent the final benefit. Developers must spend time repairing bugs and cleaning up some AI-generated code. Once this additional rework is accounted for, the average net productivity gain across industries and sectors is approximately 15–20%, with substantial variation among teams and contexts.

Q: Why can AI coding tools decrease developer productivity?

AI coding tools can reduce productivity when the code they generate creates bugs, unsuitable changes, or other problems that developers must repair. The resulting rework adds commits and visible activity without providing an equivalent amount of useful functionality. Negative outcomes are especially relevant because AI performance varies with task complexity, existing codebase constraints, language popularity, codebase size, and the maturity of the codebase.

Q: Why are commits and pull requests poor productivity metrics?

Commits and pull requests are poor standalone productivity measures because their size and value vary substantially. A developer can create more commits without delivering more useful functionality, particularly when AI-generated code introduces bugs that require additional repair tasks. Measuring increased activity alone can therefore count both the initial change and its subsequent correction as productivity, even though the team may be repeating work.

Q: How did the Stanford study measure software engineering productivity?

The researchers built a scalable model that connects to Git and analyzes the source-code changes in every commit across dimensions related to functionality, quality, maintainability, and output. Each commit has a unique author, SHA, and timestamp, enabling evaluated changes to be aggregated over time. The resulting measure focuses on what the code does rather than lines of code, commit volume, or pull-request counts.

Q: Why does AI perform better on greenfield coding tasks?

AI performs well on greenfield tasks because these assignments often involve generating boilerplate code from scratch with little existing context. In such experiments, AI-assisted developers can substantially outperform developers without AI. However, most professional engineering takes place in established codebases containing dependencies and contextual constraints, so results from simplified greenfield tasks do not transfer cleanly to typical enterprise software development.

Q: What is the difference between rework and refactoring?

Rework and refactoring both alter existing code, but rework changes code that was created much more recently. The study treats this recent correction as a stronger indication of waste because developers are revisiting work shortly after producing it. Refactoring may or may not be wasteful, while increased rework after AI adoption suggests that part of the apparent output gain is being spent fixing newly introduced problems.

Q: Can surveys accurately measure developer productivity?

Surveys were ineffective predictors of measured developer productivity in the research presented. In an experiment with 43 developers, participants estimated their position relative to global productivity levels, but their judgments differed from measured performance by about 30 percentile points. Only one in three placed themselves within the correct quartile. Surveys remain useful for morale and issues that code metrics cannot reveal, but not for productivity measurement.

Q: When should companies use AI coding tools?

Companies should use AI coding tools selectively and evaluate results within their own engineering context. The evidence indicates stronger gains for simpler tasks and greenfield work, while outcomes vary with task complexity, codebase maturity, language popularity, and codebase size. Organizations should track delivered functionality and rework after adoption because higher code volume, more commits, or positive developer perceptions do not independently prove a net productivity improvement.

Summary & Key Takeaways

• Stanford researchers analyzed software engineering activity from more than 100,000 developers across over 600 enterprise, midsized, and startup companies. The longitudinal and cross-sectional dataset includes dozens of millions of commits, billions of lines of code, and primarily private repositories, allowing productivity patterns to be examined across teams, organizations, and time periods.

• The methodology evaluates the functionality delivered by source-code changes rather than counting commits, pull requests, or lines of code. A scalable model analyzes every commit across dimensions related to output, quality, and maintainability. Its approach reflects an expert-panel evaluation process that researchers found could produce agreement and predict real engineering outcomes.

• AI initially raises coding output by roughly 30–40%, but some of that additional activity becomes rework needed to fix recently introduced code. After accounting for this waste, the average net productivity improvement is approximately 15–20%. Benefits are strongest for simpler greenfield work and vary with task complexity, codebase maturity, language popularity, and codebase size.