Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI slows down open source developers. Peter Naur can teach us why

johnwhiles.com

Post reply on HN
Print view View original

51–60 of 217 posts

Re: AI slows down open source developers. Peter Naur can teach us why

#51

Post by throwaway284927 »

sumeno wrote: post #14
rosspackard wrote: post #10

One mediocre paper/study (it should not even be called that with all the bias and sample size issues) and now we have to put up with stories re-hashing and dissecting it. I really hope these don't get upvoted more in the future. 16 devs. And they weren't allowed to pick which tasks they used the AI on. Ridiculous. Also using it on "old and >1 million line" codebases and then extrapolating that to software engineering…

> And they weren't allowed to pick which tasks they used the AI on. They were allowed to pick whether or not to use AI on a subset of tasks. They weren't forced to use AI on tasks that don't make sense for AI

That is not true, usage of AI was decided randomly. From the paper:

"To directly measure the impact of AI tools on developer productivity, we conduct a randomized controlled trial by having 16 developers complete 246 tasks (2.0 hours on average) on well-known open-source repositories (23,000 stars on average) they regularly contribute to. Each task is randomly assigned to allow or disallow AI usage, and we measure how long it takes developers to complete tasks in each condition."

Re: AI slows down open source developers. Peter Naur can teach us why

#52

Post by meindnoch »

antimora wrote: post #50

I'm one of the regular code reviewers for Burn (a deep learning framework in Rust). I recently had to close a PR because the submitter's bug fix was clearly written entirely by an AI agent. The "fix" simply muted an error instead of addressing the root cause. This is exactly what AI tends to do when it can't identify the actual problem. The code was unnecessarily verbose and even included tests for muting the error.…

>a deep learning framework in Rust [...] This is becoming a troubling trend with AI tools.

The serpent is devouring its own tail.

Re: AI slows down open source developers. Peter Naur can teach us why

#53

Post by narush »

bunderbunder wrote: post #29

> It's a really fabulous study... Ehhhh... not so much. It had serious design flaws in both the protocol and the analysis. This blog post is a fairly approachable explanation of what's wrong with it: https://www.argmin.net/p/are-developers-finally-out-of-a-job

Hey, thanks for linking this! I'm a study author, and I greatly appreciate that this author dug into the appendix and provided feedback so that other folks can read it as well.

A few notes if it's helpful:

1. This post is primarily worried about ordering considerations -- I think this is a valid concern. We explicitly call this out in the paper [1] as a factor we can't rule out -- see "Bias from issue completion order (C.2.4)". We have no evidence this occurred, but we also don't have evidence it didn't.

2. "I mean, rather than boring us with these robustness checks, METR could just release a CSV with three columns (developer ID, task condition, time)." Seconded :) We're planning on open-sourcing pretty much this data (and some core analysis code) later this week here: https://github.com/METR/Measuring-Early-2025-AI-on-Exp-OSS-D... - star if you want to dig in when it comes out.

3. As I said in my comment on the post, the takeaway at the end of the post is that "What we can glean from this study is that even expert developers aren’t great at predicting how long tasks will take. And despite the new coding tools being incredibly useful, people are certainly far too optimistic about the dramatic gains in productivity they will bring." I think this is a reasonable takeaway from the study overall. As we say in the "We do not provide evidence that:" section of the paper (Page 17), we don't provide evidence across all developers (or even most developers) -- and ofc, this is just a point-in-time measurement that could totally be different by now (from tooling and model improvements in the past month alone).

Thanks again for linking, and to the original author for their detailed review. It's greatly appreciated!

[1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf

Re: AI slows down open source developers. Peter Naur can teach us why

#54

Post by antonvs »

kevmo314 wrote: post #13
nico wrote: post #8

> They are experienced open source developers, working on their own projects I just started working on a 3-month old codebase written by someone else, in a framework and architecture I had never used before Within a couple hours, with the help of Claude Code, I had already created a really nice system to replicate data from staging to local development. Something I had built before in other projects, and I new that m…

You've missed the point of the article, which in fact agrees with your anecdote. > It's equally common for developers to work in environments where little value is placed on understanding systems, but a lot of value is placed on quickly delivering changes that mostly work. In this context, I think that AI tools have more of an advantage. They can ingest the unfamiliar codebase faster than any human can, and can often…

> You've missed the point of the article

Sadly clickbait headlines like the OP, "AI slows down open source developers," spread this misinformation, ensuring that a majority of people will have the same misapprehension.

Re: AI slows down open source developers. Peter Naur can teach us why

#55

Post by LoganDark »

meindnoch wrote: post #52
antimora wrote: post #50

I'm one of the regular code reviewers for Burn (a deep learning framework in Rust). I recently had to close a PR because the submitter's bug fix was clearly written entirely by an AI agent. The "fix" simply muted an error instead of addressing the root cause. This is exactly what AI tends to do when it can't identify the actual problem. The code was unnecessarily verbose and even included tests for muting the error.…

>a deep learning framework in Rust [...] This is becoming a troubling trend with AI tools. The serpent is devouring its own tail.

Deep learning can be incredibly cool and not just used for AI slop.

Re: AI slows down open source developers. Peter Naur can teach us why

#56

Post by bluefirebrand »

Art9681 wrote: post #38
tomasz_fm wrote: post #31

Only one developer in this study had more than 50h of Cursor experience, including time spent using Cursor during the study. That one developer saw a 25% speed improvement. Everyone else was an absolute Cursor beginner with barely any Cursor experience. I don't find it surprising that using tools they're unfamiliar with slows software engineers down. I don't think this study can be used to reach any sort of conclusio…

This is exactly my same take. Any tool an engineer is inexperienced with will slow them down. AI is no different.

This runs counter to the starry eyed promises of AI letting people with no experience accomplish things

Re: AI slows down open source developers. Peter Naur can teach us why

#57

Post by TeMPOraL »

meindnoch wrote: post #52
antimora wrote: post #50

I'm one of the regular code reviewers for Burn (a deep learning framework in Rust). I recently had to close a PR because the submitter's bug fix was clearly written entirely by an AI agent. The "fix" simply muted an error instead of addressing the root cause. This is exactly what AI tends to do when it can't identify the actual problem. The code was unnecessarily verbose and even included tests for muting the error.…

>a deep learning framework in Rust [...] This is becoming a troubling trend with AI tools. The serpent is devouring its own tail.

OTOH when they'll start getting good AI contributions, then... it'll be too late for us all.

Re: AI slows down open source developers. Peter Naur can teach us why

#58

Post by sumeno »

throwaway284927 wrote: post #51
sumeno wrote: post #14

Earlier quoted context omitted.

> And they weren't allowed to pick which tasks they used the AI on. They were allowed to pick whether or not to use AI on a subset of tasks. They weren't forced to use AI on tasks that don't make sense for AI

That is not true, usage of AI was decided randomly. From the paper: "To directly measure the impact of AI tools on developer productivity, we conduct a randomized controlled trial by having 16 developers complete 246 tasks (2.0 hours on average) on well-known open-source repositories (23,000 stars on average) they regularly contribute to. Each task is randomly assigned to allow or disallow AI usage, and we measure ho…

Directly from the paper:

> If AI is allowed, developers can use any AI tools or models they choose, including no AI tooling if they expect it to not be helpful. If AI is not allowed, no generative AI tooling can be used.

AI is allowed not required

Re: AI slows down open source developers. Peter Naur can teach us why

#59

Post by bko »

nico wrote: post #8

> They are experienced open source developers, working on their own projects I just started working on a 3-month old codebase written by someone else, in a framework and architecture I had never used before Within a couple hours, with the help of Claude Code, I had already created a really nice system to replicate data from staging to local development. Something I had built before in other projects, and I new that m…

When anecdote and data don't align, it's usually the data that's wrong.

Not always the case, but whenever I read about these strained studies or arguments about how AI is actually making people less productive, I can't help but wonder why nearly every programmer I know, myself included, finds value in these tools. I wonder if the same thing happened with higher level programming languages where people argued, you may THINK not managing your own garbage collector will lead to more productivity but actually...

Even if we weren't more "productive", millions prefer to use these tools, so it has to count for something. And I don't need a "study" to tell me that

Re: AI slows down open source developers. Peter Naur can teach us why

#60

Post by bluefirebrand »

Roscius wrote: post #33
horsawlarway wrote: post #15

Earlier quoted context omitted.

Just anecdotally - I think your reason for disagreeing is a valid statement, but not a valid counterpoint to the argument being made. So > Reason: you cannot evaluate the work accurately if you have no mental model. If there's a bug given the systems unwritten assumptions you may not catch it. This is completely correct. It's a very fair statement. The problem is that a developer coming into a large legacy project is…

> I've found that asking AI tools to generate a changeset in this case is actually a pretty solid way of starting to learn the mental model. This. Leveraging the AI to start to develop the mental model is an advantage. But, using the AI is a non-trivial skill set that needs to be learned. Skepticism of what it's saying is important. AI can be really useful just like a 747 can be useful, but you don't want someone pic…

> This. Leveraging the AI to start to develop the mental model is an advantage

Is there any evidence that AI helps you build the mental model of an unfamiliar codebase more quickly?

In my experience trying to use AI for this it often leads me into the weeds

Post reply on HN