The short version
- AI coding agents increased lines of code by 30% and commits by 20%, but did not increase the resolution rate of software issues.
- The study attributes this to a 'bottleneck' in human code review, where longer review times and more comments absorb the initial efficiency gains.
- Researchers analyzed data from over 700 firms and 700,000 employees between 2021 and 2026 to determine the impact of AI tools on engineering workflows.
A comprehensive study conducted by Harvard University researchers has challenged the assumption that artificial intelligence coding assistants directly translate into higher software productivity. While these tools are capable of generating vast amounts of functional code with remarkable speed, the research indicates that the efficiency gains achieved during the coding phase are largely neutralized by downstream constraints in the development process.
The study, led by Fiona Chen and James Stratton, utilized aggregated analytics data from Jellyfish, a platform that tracks granular engineering output. The dataset encompassed approximately 300 million individual work events, including commits and pull requests, alongside issue management software data. This information was drawn from more than 700,000 employees across over 700 software development firms, covering the period from 2021 through March 2026.
To evaluate the impact of AI integration, the researchers distinguished between AI coding assistants, which primarily aid human authors with auto-completion, and AI coding agents, which autonomously write and submit code based on prompts. By analyzing GitHub activity and directly measured usage metrics, they determined when specific companies introduced these tools into their workflows. They then employed a 'difference of differences' regression analysis to compare key variables before and after the adoption of these technologies across various organizations.
The raw data regarding code generation reveals significant increases in output volume. The introduction of AI coding agents resulted in a 30 percent increase in total lines of code generated, a 20 percent rise in the number of total commits, and a 23 percent increase in pull requests on average. These figures suggest that AI tools are indeed effective at accelerating the initial creation of code.
However, this surge in code volume did not correspond to an improvement in overall software output or feature completion. The resolution rate for Issues and Epics, which represent wholesale software features tracked by management tools like Jira, showed no statistically significant change following the introduction of AI tools. Furthermore, the researchers found no compositional shift in the size or complexity of these tracked issues, indicating that the nature of the work remained consistent despite the technological intervention.
The discrepancy between increased code generation and stagnant feature resolution is attributed to the human code review process. The study identifies this stage as a significant bottleneck. As the volume of AI-generated code increases, the time required for human reviewers to assess it also rises markedly. Pull requests are more likely to require revisions, and reviewers leave more comments, extending the duration of the review phase.
Consequently, any efficiency gained during the coding phase is absorbed by these downstream constraints. The findings suggest that firms have not seen an increase in software output or a reduction in employment as a direct result of using these AI tools. Instead, the workflow has shifted, with more time spent on reviewing and refining code rather than writing it from scratch.
These results offer a nuanced perspective on the role of AI in software development. While the technology can produce code at an unprecedented rate, the human element remains critical for ensuring accuracy and quality. The study underscores that integrating AI into engineering workflows requires careful consideration of the entire production process, not just the initial coding stage.
As companies continue to adopt AI coding agents, the findings from this Harvard study provide valuable insights into potential limitations. The lack of evidence for increased software output suggests that simply adding more code-generating capacity does not equate to faster product delivery. Organizations may need to rethink their review processes or adjust expectations regarding the immediate impact of these tools on overall productivity.
The research highlights the complexity of measuring efficiency in modern software development. It demonstrates that while AI can automate certain tasks, it also introduces new challenges in quality control and process management. The 'bottleneck' identified in human review serves as a reminder that technological advancements must be balanced with effective human oversight to achieve meaningful improvements in output.
Sources behind this briefing
Go to the original reporting
- Ars Technica↗AI coding agents generate more code, but not more software
- Forkast News↗AI Coding Agents Write 30% More Code – and Resolve Zero Additional Issues, Harvard Study Finds
