AI Tools Slow Experienced Developers by 19% in Familiar Codebases

A recent study conducted by AI research nonprofit METR has sparked significant discussion after revealing an unexpected finding: experienced software developers who used AI tools while working in familiar codebases completed their tasks 19% slower than those who did not use AI tools. The study, carried out earlier this year, involved a group of developers with extensive experience who predominantly used Cursor as the AI tool to assist them in completing tasks within open-source projects they were already familiar with.

Prior to the study, the developers believed that working with AI would accelerate their task completion, with many estimating a 24% decrease in task completion time. Even after completing the tasks with AI assistance, the developers were convinced that they had reduced the time taken to finish by 20%. However, the study's results contradicted these expectations, showing that collaborating with AI actually increased task completion time by 19%.

The study's lead authors, Joel Becker and Nate Rush, were surprised by the findings. Rush had initially expected a significant speedup in task completion, estimating a 2x increase in efficiency. This widespread belief that AI enhances human productivity has led several companies to invest heavily in AI products aimed at aiding software development. However, the METR study has highlighted that this assumption does not hold true in all software development scenarios, particularly for experienced developers working with large, established open-source codebases.

The study's authors noted that other research often relies on software development benchmarks for AI, which sometimes misrepresent real-world tasks. The slowdown experienced by the developers was attributed to the time spent reviewing and correcting suggestions made by AI models. Despite these findings, the majority of the study's participants, as well as the authors, continue to use Cursor. They cited the ease of use and the reduced feeling of it being a chore as reasons for their continued use of the tool.

Despite the study's findings, the fear of AI replacing human jobs persists, with some experts predicting that AI could replace entry-level coding positions in the next one to five years. This sentiment is shared by Dario Amodei, CEO of Anthropic, who believes that the shift towards AI in programming careers is already underway. The tech industry has seen significant layoffs this year, with major companies like Microsoft, Google, Salesforce, Intel, and IBM citing a shift towards AI as a reason for reducing their workforce. Microsoft, for instance, has dismissed up to 9,000 employees, with 40% of the recent layoffs targeting software engineers whose tasks were outsourced to AI tools. Google has also laid off hundreds of workers following its investment in AI startup Anthropic, though not all dismissed workers were replaced with AI. Salesforce has cut 1,000 jobs this year to focus on AI roles, Intel has reduced its workforce by 15,000, and IBM plans to replace 30% of its back-office roles with AI by 2030.

Despite these layoffs, many tech professionals view AI as a collaborator rather than a threat. The enthusiasm of the METR study's authors and participants to continue using AI tools, despite the slowdown in task completion, indicates a growing acceptance of AI as a valuable tool in software development. The study's findings suggest that while AI may not always speed up task completion, it can make the development process easier and more enjoyable, which may be a significant factor in its continued adoption.