AI Coding Assistants May Not Boost Productivity As Expected

OpenAI's plans to acquire Windsurf, a startup specializing in AI software for coding, for $3 billion fell through. Instead, Google hired Windsurf's CEO Varun Mohan and cofounder Douglas Chen, along with other staff, and licensed Windsurf’s technology. This deal mirrors several other arrangements where big tech companies acquire AI startups without a full acquisition, such as Meta’s deal with Scale AI and Microsoft’s deal with Inflection. The increasingly complex relationship between OpenAI and Microsoft is notable, as is the structure of these non-acquisition deals, which seem to avoid legal challenges from regulators or venture backers.

While many agree that coding is a clear use case for generative AI, recent studies complicate this view. A nonprofit called METR conducted a randomized control trial involving 16 developers to see if using the code editor Cursor Pro, integrated with Anthropic’s Claude Sonnet 3.5 and 3.7 models, improved their productivity. Developers estimated that using AI would make them 24% faster, but the trial found that it actually took them 19% longer to finish tasks. The developers found that Cursor could not reliably generate code as good as theirs and spent significant time reviewing and cleaning up AI-generated outputs. This cognitive burden, however, made the task mentally easier for the developers, who continued to use Cursor despite the time it took to edit the code.

Another study from Harvard Business School and Microsoft found that developers using GitHub Copilot spent more time on coding and less on project management tasks. This allowed them to work independently and explore possible solutions more thoroughly, even if it didn’t necessarily save time. A third study from Modelbest, BUPT, Tsinghua University, and the University of Sydney found that using multiple large language models to take on specific roles in the software development process improved results. This architecture, called “ChatDev,” suggests that the problem with AI coding assistants might be how they are used rather than the technology itself. However, building teams of AI agents to work in this way uses up a lot more computing power, which can be expensive.

These studies raise questions about the perceived return on investment from AI coding assistants. While developers may find AI tools mentally easier to use, the actual time savings and efficiency gains are not as clear-cut as initially thought. The perceived benefits of AI coding assistants might be more about the cognitive ease they provide rather than actual productivity improvements. This suggests that the ROI from AI coding assistants could be a mirage, and more research is needed to fully understand their impact on software development.