Measuring the impact of AI on experienced open-source developer productivity • Hacker News Board

Re: Measuring the impact of AI on experienced open-source developer productivity

#21

Post by 30minAdayHN » Thu, Jul 10, 2025, 5:24 PM UTC

This study focused on experienced OSS maintainers. Here is my personal experience, but a very different persona (or opposite to the one in the study). I always wanted to contribute to OSS but never had time to. Finally was able to do that, thanks to AI. Last month, I was able to contribute to 4 different repositories which I would never have dreamed of doing it. I was using an async coding agent I built[1], to generate PRs given a GitHub issue. Some PRs took a lot of back and forth. And some PRs were accepted as is. Without AI, there is no way I would have contributed to those repositories.

One thing that did work in my favor is that, I was clearly creating a failing repro test case, and adding before and after along with PR. That helped getting the PR landed.

There are also a few PRs that never got accepted because the repro is not as strong or clear.

[1] https://workback.ai

Re: Measuring the impact of AI on experienced open-source developer productivity

#22

Post by reaperducer » Thu, Jul 10, 2025, 5:26 PM UTC

fritzo wrote: post #8

As an open source maintainer on the brink of tech debt bankruptcy, I feel like AI is a savior, helping me keep up with rapid changes to dependencies, build systems, release methodology, and idioms.

[flagged]

Re: Measuring the impact of AI on experienced open-source developer productivity

#23

Post by MYEUHD » Thu, Jul 10, 2025, 5:26 PM UTC

This does not mention the open-source developer time wasted while reviewing vibe coded PRs

Re: Measuring the impact of AI on experienced open-source developer productivity

#24

Post by castratikron » Thu, Jul 10, 2025, 5:26 PM UTC

I really like those graphics, does anyone know the tool was used to create them?

Re: Measuring the impact of AI on experienced open-source developer productivity

#25

Post by fritzo » Thu, Jul 10, 2025, 5:27 PM UTC

aerhardt wrote: post #14

fritzo wrote: post #8

As an open source maintainer on the brink of tech debt bankruptcy, I feel like AI is a savior, helping me keep up with rapid changes to dependencies, build systems, release methodology, and idioms.

But what about producing actual code?

Producing code is overrated. There's lots of old code whose lifetime we can extend.

Re: Measuring the impact of AI on experienced open-source developer productivity

#26

Post by eknkc » Thu, Jul 10, 2025, 5:28 PM UTC

noisy_boy wrote: post #6

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

It works great on adding stuff to an already established codebase. Things like “we have these search parameters, also add foo”. Remove anything related to x…

Re: Measuring the impact of AI on experienced open-source developer productivity

#27

Post by threetonesun » Thu, Jul 10, 2025, 5:28 PM UTC

emodendroket wrote: post #18

noisy_boy wrote: post #6

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

I think it’s most useful when you basically need Stack Overflow on steroids: I basically know what I want to do but I’m not sure how to achieve it using this environment. It can also be helpful for debugging and rubber ducking generally.

Absolutely this. For a while I was working with a language I was only partially familiar with, and I'd say "here's how I would do this in [primary language], rewrite it in [new language]" and I'd get a decent piece of code back. A little searching in the project to make sure it was stylistically correct and then done.

Re: Measuring the impact of AI on experienced open-source developer productivity

#28

Post by narush » Thu, Jul 10, 2025, 5:28 PM UTC

Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible!

If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper.

[1] https://x.com/METR_Evals/status/1943360399220388093

Re: Measuring the impact of AI on experienced open-source developer productivity

#29

Post by iLoveOncall » Thu, Jul 10, 2025, 5:30 PM UTC

NewsaHackO wrote: post #17

So they paid developers 300 x 246 = about 73K just for developer recruitment for the study, which is not in any academic journal, or has no peer reviews? The underlying paper looks quite polished and not overtly AI generated so I don't want to say it entirely made up, but how were they even able to get funding for this?

https://metr.org/about Seems like they get paid by AI companies, and they also get government funding.

Re: Measuring the impact of AI on experienced open-source developer productivity

#30

Post by incomingpain » Thu, Jul 10, 2025, 5:30 PM UTC

Essentially an advertisement against Cursor Pro and/or Claude Sonnet 3.5/3.7

I think personally when i tried tools like Void IDE, I was fighting with Void too much. It is beta software, it is buggy, but also the big one... learning curve of the tool.

I havent had the chance to try cursor but i imagine its going to have a learning curve as a new tool.

So perhaps there is a slowdown at first expected; but later after you get your context and prompting down pat. Asking specifically for what you want. Then you get your speed up.