Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

201–210 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#201

Post by nalllar »

blibble wrote: post #169
steveklabnik wrote: post #151

Earlier quoted context omitted.

> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

If you interact with internet comments and discussions as an amorphous blob of people you'll see a constant trickle of the view that models now are useful, and before were useless.

If you pay attention to who says it, you'll find that people have different personal thresholds for finding llms useful, not that any given person like steveklabnik above keeps flip-flopping on their view.

This is a variant on the goomba fallacy: https://englishinprogress.net/gen-z-slang/goomba-fallacy-exp...

Re: Measuring the impact of AI on experienced open-source developer productivity

#202

Post by mattmanser »

steveklabnik wrote: post #185
blibble wrote: post #169

Earlier quoted context omitted.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

Sorry, that’s not my take. I didn’t think these tools were useful until the latest set of models, that is, they crossed the threshold of usefulness to me. Even then though, “technology gets better over time” shouldn’t be surprising, as it’s pretty common.

Do you really see a massive jump?

For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend.

I'd say they're only marginally better today than they were even 2 years ago.

Every time a new model comes out you get a bunch of people raving how great the new one is and I honestly can't really tell the difference. The only real difference is reasoning models actually slowed everything down, but now I see its reasoning. It's only useful because I often spot it leaving out important stuff from the final answer.

Re: Measuring the impact of AI on experienced open-source developer productivity

#203

Post by lmeyerov »

As someone has been doing hardcore genai for 2+ years, my experience has been, and what we advise internally:

* 3 weeks to transition from ai pairing to AI Delegation to ai multitasking. So work gains are mostly week 3+. That's 120+ hours in, as someone pretty senior here.

* Speedup is the wrong metric. Think throughput, not latency. Some finite amount of work might take longer, but the volume of work should go up because AI can do more on a task and diff tasks/projects in parallel.

Both perspectives seem consistent with the paper description...

Re: Measuring the impact of AI on experienced open-source developer productivity

#204

Post by Ntrails »

cwillu wrote: post #180
simonw wrote: post #171

Earlier quoted context omitted.

The previous model retroactively becomes not as good as the best available models. I don't think that's a huge surprise.

The surprise is the implication that the crossover between net-negative and net-positive impact happened to be in the last 4 months, in light of the initial release 2 years ago and sufficient public attention for a study to be funded and completed. Yes, it might make a difference, but it is a little tiresome that there's always a “this is based on a model that is x months old!” comment, because it will always be true…

Some of it is just that (probably different) people said the same damn things 6 months ago.

"No, the 2.8 release is the first good one. It massively improves workflows"

Then, 6 months later, the study comes out.

"Ah man, 2.8 was useless, 3.0 really crossed the threshold on value add"

At some point, you roll your eyes and assume it is just snake oil sales

Re: Measuring the impact of AI on experienced open-source developer productivity

#205

Post by Terr_ »

pdabbadabba wrote: post #184
blibble wrote: post #169

Earlier quoted context omitted.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

Maybe it's convenient. But isn't it also just a fact that some of the models available today are better than the ones available five months ago?

That's not the issue. Their complaint is that proponents keep revising what ought to be fixed goalposts... Well, fixed unless you believe unassisted human developers are also getting dramatically better at their jobs every year.

Like the boy who cried wolf, it'll eventually be true with enough time... But we should stop giving them the benefit of the doubt.

_____

Jan 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious."

Feb 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious."

Mar 2025: "Ignore last month's models, they aren't good enough to show a marked increase in human productivity, test with this month's models and the benefits are obvious."

Apr 2025: [Ad nauseam, you get the idea]

Re: Measuring the impact of AI on experienced open-source developer productivity

#206

Post by ipaddr »

steveklabnik wrote: post #185
blibble wrote: post #169

Earlier quoted context omitted.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

Sorry, that’s not my take. I didn’t think these tools were useful until the latest set of models, that is, they crossed the threshold of usefulness to me. Even then though, “technology gets better over time” shouldn’t be surprising, as it’s pretty common.

Wait until the next set. You will find you the previous ones weren't useful after all.

Re: Measuring the impact of AI on experienced open-source developer productivity

#207

Post by hombre_fatal »

mattmanser wrote: post #202
steveklabnik wrote: post #185

Earlier quoted context omitted.

Sorry, that’s not my take. I didn’t think these tools were useful until the latest set of models, that is, they crossed the threshold of usefulness to me. Even then though, “technology gets better over time” shouldn’t be surprising, as it’s pretty common.

Do you really see a massive jump? For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend. I'd say they're only marginally better today than they were even 2 years ago. Every time a new model c…

I see a massive jump every time.

Just two years ago, this failed.

> Me: What language is this: "esto está escrito en inglés"

> LLM: English

Gemini and Opus have solved questions that took me weeks to solve myself. And I'll feed some complex code into each new iteration and it will catch a race condition I missed even with testing and line by line scrutiny.

Consider how many more years of experience you need as a software engineer to catch hard race conditions just from reading code than someone who couldn't do it after trying 100 times. We take it for granted already since we see it as "it caught it or it didn't", but these are massive jumps in capability.

Re: Measuring the impact of AI on experienced open-source developer productivity

#208

Post by ieie3366 »

LLMs are godtier if you know what you’re doing, and prompt them with ”do X”, where x is a SELF-CONTAINED change you would manually know how to implement

For example, today I asked claude to implement per-user rate-limiting into my nestjs service, then iterated by asking implementing specific unit tests and some refactoring. It one-shot everything. I would say 90% time savings.

Unskilled people ask them ”i have giant problem X solve it” and end up with slop

Re: Measuring the impact of AI on experienced open-source developer productivity

#209

Post by igorkraw »

narush wrote: post #107
igorkraw wrote: post #86

Earlier quoted context omitted.

Could you either release the dataset (raw but anonymized) for independent statistical évaluation or at least add the absolute times of each dev per task to the paper? I'm curious what the absolute times of each dev with/without AI was and whether the one guy with lots of Cursor experience was actually faster than the rest of just a slow typer getting a big boost out of llms Also, cool work, very happy to see actually…

Yep, sorry, meant to post this somewhere but forgot in final-paper-polishing-sprint yesterday! We'll be releasing anonymized data and some basic analysis code to replicate core results within the next few weeks (probably next, depending). Our GitHub is here ( http://github.com/METR/ ) -- or you can follow us ( https://x.com/metr_evals ) and we'll probably tweet about it.

Cool, thanks a lot. Btw, I have a very tiny tiny (50 to 100 audience ) podcast where we try to give context to what we call the "muck" of AI discourse (trying to ground claims into both what we would call objectively observable facts/évidence, and then _separately_ giving out own biased takes), if you would be interested to come on it and chat => contact email in my profile.

Re: Measuring the impact of AI on experienced open-source developer productivity

#210

Post by thepasswordis »

I actually think that pasting questions into chatGPT etc. and then getting general answers to put into your code is the way.

“One shotting” apps, or even cursor and so forth seem like a waste of time. It feels like if you prompt it just right it might help but then it never really does.

Post reply on HN