Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

11–20 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#11

Post by IshKebab »

I wonder if the discrepancy is that it felt like it was taking less time because they were having to do less thinking which feels like it is easier and hence faster.

Even so... I still would be really surprised if there wasn't some systematic error here skewing the results, like the developers deliberately picked "easy" tasks that they already knew how to do, so implementing them themselves was particularly fast.

Seems like they authors had about as good methodology as you can get for something like this. It's just really hard to test stuff like this. I've seen studies proving that code comments don't matter for example... are you going to stop writing comments? No.

Re: Measuring the impact of AI on experienced open-source developer productivity

#12

Post by IshKebab »

barbazoo wrote: post #10
IshKebab wrote: post #5

Earlier quoted context omitted.

They paid the developers about $75k in total to do this so I wouldn't hold your breath!

That's a lot of money for many of us. Do you know those folks were in a HCOL area?

No idea. They don't say who they were; just random popular GitHub projects.

To be clear it wasn't $75k each.

Re: Measuring the impact of AI on experienced open-source developer productivity

#13

Post by add-sub-mul-div »

dash2 wrote: post #4

The authors say "High developer familiarity with repositories" is a likely reason for the surprising negative result, so I wonder if this generalizes beyond that.

Without the familiarity would the work be getting done effectively? What does it mean for someone to commit AI code that they can't fully understand?

Re: Measuring the impact of AI on experienced open-source developer productivity

#14

Post by aerhardt »

fritzo wrote: post #8

As an open source maintainer on the brink of tech debt bankruptcy, I feel like AI is a savior, helping me keep up with rapid changes to dependencies, build systems, release methodology, and idioms.

But what about producing actual code?

Re: Measuring the impact of AI on experienced open-source developer productivity

#15

Post by wmeredith »

noisy_boy wrote: post #6

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

> and then you spend 80% of the time to get the rest of the 20% done

This was my pr-AI experience anyway, so getting that first chunk of time back is helpful.

Related: One of the better takes I've seen on AI from an experienced developer was, "90% of my skills just became worthless, and the other 10% just became 1,000 times more valuable." There's some hyperbole there, I but I like the gist.

Re: Measuring the impact of AI on experienced open-source developer productivity

#17

Post by NewsaHackO »

So they paid developers 300 x 246 = about 73K just for developer recruitment for the study, which is not in any academic journal, or has no peer reviews? The underlying paper looks quite polished and not overtly AI generated so I don't want to say it entirely made up, but how were they even able to get funding for this?

Re: Measuring the impact of AI on experienced open-source developer productivity

#18

Post by emodendroket »

noisy_boy wrote: post #6

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

I think it’s most useful when you basically need Stack Overflow on steroids: I basically know what I want to do but I’m not sure how to achieve it using this environment. It can also be helpful for debugging and rubber ducking generally.
Post reply on HN