Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

291–300 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#291

Post by ryanar »

igorkraw wrote: post #209
narush wrote: post #107

Earlier quoted context omitted.

Yep, sorry, meant to post this somewhere but forgot in final-paper-polishing-sprint yesterday! We'll be releasing anonymized data and some basic analysis code to replicate core results within the next few weeks (probably next, depending). Our GitHub is here ( http://github.com/METR/ ) -- or you can follow us ( https://x.com/metr_evals ) and we'll probably tweet about it.

Cool, thanks a lot. Btw, I have a very tiny tiny (50 to 100 audience ) podcast where we try to give context to what we call the "muck" of AI discourse (trying to ground claims into both what we would call objectively observable facts/évidence, and then _separately_ giving out own biased takes), if you would be interested to come on it and chat => contact email in my profile.

podcast link?

Re: Measuring the impact of AI on experienced open-source developer productivity

#292

Post by malfist »

lukan wrote: post #61
skydhash wrote: post #48

Earlier quoted context omitted.

The issue is that it is slow and verbose, at least in its default configuration. The amount of reading is non trivial. There’s a reason most references are dense.

Those issues you can partly solve by changing the prompt to tell it to be concise and don't explain its code. But nothing will make them stick to the one API version I use.

The less verbosity you allow the dumber the LLM is. It thinks in tokens and if you keep it from using tokens it's lobotomized.

Re: Measuring the impact of AI on experienced open-source developer productivity

#293

Post by vidarh »

mattmanser wrote: post #202
steveklabnik wrote: post #185

Earlier quoted context omitted.

Sorry, that’s not my take. I didn’t think these tools were useful until the latest set of models, that is, they crossed the threshold of usefulness to me. Even then though, “technology gets better over time” shouldn’t be surprising, as it’s pretty common.

Do you really see a massive jump? For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend. I'd say they're only marginally better today than they were even 2 years ago. Every time a new model c…

I've gone from asking the tools how to do things, and cut and pasting the bits (often small) that'd be helpful, via using assistants that I'd review every decision of and often having to start over, to now often starting an assistant with broad permissions and just reviewing the diff later, after they've made the changes pass the test suite, run a linter and fixed all the issues it brought up, and written a draft commit message.

The jump has been massive.

Re: Measuring the impact of AI on experienced open-source developer productivity

#294

Post by atlintots »

leshow wrote: post #272
thesz wrote: post #251

Earlier quoted context omitted.

> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This is what I heard about strong type systems (especially Haskell's) about 20-15 years ago. "History does not repeat, but it rhymes." If we rhyme "strong types will change the world" with "agentic LLMs will change the world," what do we get? My personal the…

I don't think that's a fair comparison. Type systems don't produce probabilistic output. Their entire purpose is to reduce the scope of possible errors you can write. They kind of did change the world, didn't they? I mean, not everyone is writing Haskell but Rust exists and it's doing pretty well. There was also not really a case to be made where type systems made software in general _worse_. But you could definitely…

Its too bad the management people never pushed Haskell as hard as they're pushing AI today! Alas.

Re: Measuring the impact of AI on experienced open-source developer productivity

#295

Post by bluefirebrand »

pdabbadabba wrote: post #249
Terr_ wrote: post #205

Earlier quoted context omitted.

That's not the issue. Their complaint is that proponents keep revising what ought to be fixed goalposts... Well, fixed unless you believe unassisted human developers are also getting dramatically better at their jobs every year. Like the boy who cried wolf, it'll eventually be true with enough time... But we should stop giving them the benefit of the doubt. _____ Jan 2025: "Ignore last month's models, they aren't goo…

Fair enough. For what it's worth, I've always thought that the more reasonable claim is that AI tools make poor-average developers more productive, not necessarily expert developers.

Personally I don't want poor-average developers to be more productive, I want them to be more expert

Re: Measuring the impact of AI on experienced open-source developer productivity

#296

Post by malfist »

longwave wrote: post #193
pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

I also wonder how many of the numerous AI proponents in HN comments are subject to the same effect. Unless they are truly measuring their own performance, is AI really making them more productive?

How would you even measure your own performance? You can go and redo something, forgetting everything you did along the way the first time

Re: Measuring the impact of AI on experienced open-source developer productivity

#297

Post by mwigdahl »

blibble wrote: post #169
steveklabnik wrote: post #151

Earlier quoted context omitted.

> Current LLMs One thing that happened here is that they aren't using current LLMs: > Most issues were completed in February and March 2025, before models like Claude 4 Opus or Gemini 2.5 Pro were released. That doesn't mean this study is bad! In fact, I'd be very curious to see it done again, but with newer models, to see if that has an impact.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

I've been a proponent for a long time, so I certainly fit this at least partially. However, the combination of Claude Code and the Claude 4 models has pushed the response to my demos of AI coding at my org from "hey, that's kind of cool" to "Wow, can you get me an API key please?"

It's been a very noticeable uptick in power, and although there have been some nice increases with past model releases, this has been both the largest and the one that has unlocked the most real value since I've been following the tech.

Re: Measuring the impact of AI on experienced open-source developer productivity

#298

Post by lll-o-lll »

EarthLaunch wrote: post #239
evanelias wrote: post #233

Earlier quoted context omitted.

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If…

> The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con https://softwarecrisis.dev/letters/llmentalist/ Plus there's a gambling mechanic: Push the button, sometimes get things for free.

This is very interesting and disturbing. We are outsourcing our decision making to an algorithmic “Mentalist” and will reap a terrible reward. I need to ween myself off the comforting teat of the chatbot psychic.

Re: Measuring the impact of AI on experienced open-source developer productivity

#299

Post by achierius »

mwigdahl wrote: post #297
blibble wrote: post #169

Earlier quoted context omitted.

> One thing that happened here is that they aren't using current LLMs I've been hearing this for 2 years now the previous model retroactively becomes total dogshit the moment a new one is released convenient, isn't it?

I've been a proponent for a long time, so I certainly fit this at least partially. However, the combination of Claude Code and the Claude 4 models has pushed the response to my demos of AI coding at my org from "hey, that's kind of cool" to "Wow, can you get me an API key please?" It's been a very noticeable uptick in power, and although there have been some nice increases with past model releases, this has been both…

Is that really the case vs. 3.7? For me that was the threshold, and since then the improvements have been nice but not as significant.

Re: Measuring the impact of AI on experienced open-source developer productivity

#300

Post by jwrallie »

evanelias wrote: post #233
pera wrote: post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If…

Like the feeling of the command line being always faster than using the GUI? Different ways we engage with a task can change our time perception.

I wish there was a simple way to measure energy spent instead of time. Maybe nature is just optimizing for something else.

Post reply on HN