Measuring the impact of AI on experienced open-source developer productivity
1–10 of 501 posts
Re: Measuring the impact of AI on experienced open-source developer productivity
#2Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.
Re: Measuring the impact of AI on experienced open-source developer productivity
#3> developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.
I feel like there are two challenges causing this. One is that it's difficult to get good data on how long the same person in the same context would have taken to do a task without AI vs with. The other is that it's tempting to time an AI with metrics like how long until the PR was opened or merged. But the AI workflow fundamentally shifts engineering hours so that a greater percentage of time is spent on refactoring, testing, and resolving issues later in the process, including after the code was initially approved and merged. I can see how it's easy for a developer to report that AI completed a task quickly because the PR was opened quickly, discounting the amount of future work that the PR created.
Re: Measuring the impact of AI on experienced open-source developer productivity
#4The authors say "High developer familiarity with repositories" is a likely reason for the surprising negative result, so I wonder if this generalizes beyond that.
Re: Measuring the impact of AI on experienced open-source developer productivity
#5Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.
They paid the developers about $75k in total to do this so I wouldn't hold your breath!
Re: Measuring the impact of AI on experienced open-source developer productivity
#6It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up.
I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebbles such as that small bit of syntax I forgot, basically to keep up the velocity. However, I don't look at the wholesale code it offers. I think keeping the active thinking cap on results in code I actually understand while avoiding skill atrophy.
Re: Measuring the impact of AI on experienced open-source developer productivity
#7The authors say "High developer familiarity with repositories" is a likely reason for the surprising negative result, so I wonder if this generalizes beyond that.
Like if it generalizes to situations where the developer is not familiar with the repo? That doesn’t seem like generalizing, that seems like specifying. Am I wrong in saying that the majority of developer time is spent in repos that they’re familiar with? Every job and project I’ve worked has been on a fixed set of repos the entire time. If AI is only helpful for the first week or two on a project, that’s not very many cases it’s useful for.
Re: Measuring the impact of AI on experienced open-source developer productivity
#8As an open source maintainer on the brink of tech debt bankruptcy, I feel like AI is a savior, helping me keep up with rapid changes to dependencies, build systems, release methodology, and idioms.
Re: Measuring the impact of AI on experienced open-source developer productivity
#9> developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I feel like there are two challenges causing this. One is that it's difficult to get good data on how long the same person in the same context would have taken to do a task without AI vs with. The other is that it's tempting to time an AI with metrics like how long until the PR w…
It's really hard to attribute productivity gains/losses to specific technologies or practices, I'm very wary of self-reported anecdotes in any direction precisely because it's so easy to fool ourselves.
I'm not making any claim in either direction, the authors themselves recognize the study's limitations, I'm just trying to say that everyone should have far greater error bars. This technology is the weirdest shit I've seen in my lifetime, making deductions about productivity from anecdotes and dubious benchmarks is basically reading tea leaves.
Re: Measuring the impact of AI on experienced open-source developer productivity
#10Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.
They paid the developers about $75k in total to do this so I wouldn't hold your breath!
That's a lot of money for many of us. Do you know those folks were in a HCOL area?