Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.
They paid the developers about $75k in total to do this so I wouldn't hold your breath!
Neat, how to sign up??
Measuring the impact of AI on experienced open-source developer productivity
41–50 of 501 posts
Re: Measuring the impact of AI on experienced open-source developer productivity
#42> We pay developers $150/hr as compensation for their participation in the study.
Can someone point me to these 300k/yr jobs?
Re: Measuring the impact of AI on experienced open-source developer productivity
#43So they paid developers 300 x 246 = about 73K just for developer recruitment for the study, which is not in any academic journal, or has no peer reviews? The underlying paper looks quite polished and not overtly AI generated so I don't want to say it entirely made up, but how were they even able to get funding for this?
Most of the world provides funding for research, the US used to provide funding but now that has been mostly gutted.
Re: Measuring the impact of AI on experienced open-source developer productivity
#44It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…
well we used to have a sort of inverse pareto where 80% of the work took 80% of the effort and the remaining 20% of the work also took 80% of the effort. I do think you're onto something with getting pebbles out of the road inasmuch as once I know what I need to do AI coding makes the doing much faster. Just yesterday I was playing around with removing things from a List object using the Java streams API and I kept r…
Code reuse at scale: 80 + 80 = 160% ~ phi...coincidence?
I think this may become a long horizon harvest for the rigorous OOP strategy, may Bill Joy be disproved.
Gray goo may not [taste] like steel-cut oatmeal.
Re: Measuring the impact of AI on experienced open-source developer productivity
#45One thing I could not find on a cursory read is how used were those developers to AI tools. I would expect someone using those regularly to benefit while someone who only played with them a couple of time would likely be slowed down as they deal with the friction of learning to be productive with the tool.
Re: Measuring the impact of AI on experienced open-source developer productivity
#46It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…
As an old dev this is really all I want: a sort of autocorrect for my syntactical errors to save me a couple compile-edit cycles.
Re: Measuring the impact of AI on experienced open-source developer productivity
#47So they paid developers 300 x 246 = about 73K just for developer recruitment for the study, which is not in any academic journal, or has no peer reviews? The underlying paper looks quite polished and not overtly AI generated so I don't want to say it entirely made up, but how were they even able to get funding for this?
Our largest funding was through The Audacious Project -- you can see an announcement here: https://metr.org/blog/2024-10-09-new-support-through-the-aud... Per our website, “To date, April 2025, we have not accepted compensation from AI companies for the evaluations we have conducted.” You can check out the footnote on this page: https://metr.org/donate
This is really disingenuous when you also say that OpenAI and Anthropic have provided you with access and compute credits (on https://metr.org/about).
Not all payment is cash. Compute credits is still by all means compensation.
Re: Measuring the impact of AI on experienced open-source developer productivity
#48It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…
I think it’s most useful when you basically need Stack Overflow on steroids: I basically know what I want to do but I’m not sure how to achieve it using this environment. It can also be helpful for debugging and rubber ducking generally.
The issue is that it is slow and verbose, at least in its default configuration. The amount of reading is non trivial. There’s a reason most references are dense.
Re: Measuring the impact of AI on experienced open-source developer productivity
#49AI by design can only repeat and recombine past material. Therefore actual invention is out.
Re: Measuring the impact of AI on experienced open-source developer productivity
#50It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…
> and then you spend 80% of the time to get the rest of the 20% done This was my pr-AI experience anyway, so getting that first chunk of time back is helpful. Related: One of the better takes I've seen on AI from an experienced developer was, "90% of my skills just became worthless, and the other 10% just became 1,000 times more valuable." There's some hyperbole there, I but I like the gist.
It’s not funny when you find yourself redoing the first 80%, as the only way to complete the second 80%.