Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

241–250 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#241

Post by ummonk »

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

Devil's advocate: it's also possible the one developer hasn't become more productive with Cursor, but rather has atrophied their non-AI productivity due to becoming reliant on Cursor.

Re: Measuring the impact of AI on experienced open-source developer productivity

#242

Post by csherratt »

evanelias wrote: post #233
pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If…

That's my suspicion to.

My issue with this being a 'negative' thing is that I'm not sure it is. It works off of the same hunting / foraging instincts that keep us alive. If you feel addiction to something positive, it is bad?

Social media is negative because it addicts you to mostly low quality filler content. Content that doesn't challenge you. You are reading shit posts instead of reading a book or doing something with better for you in the long run.

One could argue that's true for AI, but I'm not confident enough to make such a statement.

Re: Measuring the impact of AI on experienced open-source developer productivity

#243

Post by Dzugaru »

furyofantares wrote: post #178
bluefirebrand wrote: post #133

Earlier quoted context omitted.

> Take a whole week to do a month's worth of features Everything else in your post is so reasonable and then you still somehow ended up suggesting that LLMs should be quadrupling our output

I'm specifically talking about greenfield work. I do a lot of game prototypes, it definitely does that at the very beginning.

This is really interesting, because I do gamejams from time to time - and I try every time to make it work, but I'm still quite a lot faster doing stuff myself.

This is visible under extreme time pressure of producing a working game in 72 hours (our team scores consistenly top 100 in Ludum Dare which is a somewhat high standard).

We use a popular Unity game engine all LLMs have wealth of experience (as in game development in general), but the output is 80% so strangely "almost correct but not usable" that I cannot take the luxury of letting it figure it out, and use it as fancy autocomplete. And I also still check docs and Stackoverflow-style forums a lot, because of stuff it plainly mades up.

One of the reasons is maybe our game mechanics often is a bit off the beaten road, though the last game we made was literally a platformer with rope physics (LLM could not produce a good idea how to make stable and simple rope physics under our constraints codeable in 3 hours time).

Re: Measuring the impact of AI on experienced open-source developer productivity

#244

Post by jona777than »

furyofantares wrote: post #72
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. I totally agree with this. Although also, you can end up in a bad spot even after you've gotten pretty good at getting the AI tools to give you good output, because you fail to learn the code you're producing well. A developer gets better at the code they're…

> but then hit a brick wall

This is my intuition as well. I had a teammate use a pretty good analogy today. He likened vibe coding to vacuuming up a string in four tries when it only takes one try to reach down and pick it up. I thought that aligned well with my experience with LLM assisted coding. We have to vacuum the floor while exercising the "difficult skill [of] continually avoiding temptation to vibe"

Re: Measuring the impact of AI on experienced open-source developer productivity

#245

Post by Filligree »

keeda wrote: post #211
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. Yes, and I'll add that there is likely no single "golden workflow" that works for everybody, and everybody needs to figure it out for themselves. It took me months to figure out how to be effective with these tools, and I doubt my approach will transfer over…

> I've not worked with coding agents, but I doubt this approach will transfer over well to them.

Actually, it works well so long as you tell them when you’ve made a change. Claude gets confused if things randomly change underneath it, but it has no trouble so long as you give it a short explanation.

Re: Measuring the impact of AI on experienced open-source developer productivity

#246

Post by thesz »

What is interesting here is that all predictions were positive, but results are negative.

This shows that everyone in the study (economic experts, ML experts and even developers themselves, even after getting experience) are novices if we look at them from the Dunning-Kruger effect [1] perspective.

[1] https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

"The Dunning–Kruger effect is a cognitive bias in which people with limited competence in a particular domain overestimate their abilities."

Re: Measuring the impact of AI on experienced open-source developer productivity

#247

Post by mattl »

I don't understand how anyone doing open source can use something trained on other people's code as a tool for contributions.

I wouldn't accept someone's copy and pasted code from another project if it were under an incompatible license, let alone something with unknown origin.

Re: Measuring the impact of AI on experienced open-source developer productivity

#248

Post by evanelias »

csherratt wrote: post #242
evanelias wrote: post #233

Earlier quoted context omitted.

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If…

That's my suspicion to. My issue with this being a 'negative' thing is that I'm not sure it is. It works off of the same hunting / foraging instincts that keep us alive. If you feel addiction to something positive, it is bad? Social media is negative because it addicts you to mostly low quality filler content. Content that doesn't challenge you. You are reading shit posts instead of reading a book or doing something…

The study found AI caused a "significant slowdown" in developer efficiency though, so that doesn't seem positive!

Re: Measuring the impact of AI on experienced open-source developer productivity

#249

Post by pdabbadabba »

Terr_ wrote: post #205
pdabbadabba wrote: post #184

Earlier quoted context omitted.

Maybe it's convenient. But isn't it also just a fact that some of the models available today are better than the ones available five months ago?

That's not the issue. Their complaint is that proponents keep revising what ought to be fixed goalposts... Well, fixed unless you believe unassisted human developers are also getting dramatically better at their jobs every year. Like the boy who cried wolf, it'll eventually be true with enough time... But we should stop giving them the benefit of the doubt. _____ Jan 2025: "Ignore last month's models, they aren't goo…

Fair enough. For what it's worth, I've always thought that the more reasonable claim is that AI tools make poor-average developers more productive, not necessarily expert developers.

Re: Measuring the impact of AI on experienced open-source developer productivity

#250

Post by jstummbillig »

card_zero wrote: post #213
jstummbillig wrote: post #194

Earlier quoted context omitted.

Convenient for whom and what...? There is nothing tangible to gain from you believing or not believing that someone else does (or does not) get a productivity boost from AI. This is not a religion and it's not crypto. The AI users' net worth is not tied to another ones use of or stance on AI (if anything, it's the opposite). More generally, the phenomenon this is quite simply explained and nothing surprising: New thi…

I saw that edit. Indeed you can't predict that rejecting a new thing is part of a routine of being wrong. It's true that "it's strange and new, therefore I hate it" is a very human (and adorable) instinct, but sometimes it's reasonable.

"I saw that edit" lol
Post reply on HN