I really like those graphics, does anyone know the tool was used to create them?
The graphs are all matplotlib. The methodology figure is built in Figma! (Source: I'm a paper author :)).
Measuring the impact of AI on experienced open-source developer productivity
91–100 of 501 posts
Re: Measuring the impact of AI on experienced open-source developer productivity
#92Earlier quoted context omitted.
well we used to have a sort of inverse pareto where 80% of the work took 80% of the effort and the remaining 20% of the work also took 80% of the effort. I do think you're onto something with getting pebbles out of the road inasmuch as once I know what I need to do AI coding makes the doing much faster. Just yesterday I was playing around with removing things from a List object using the Java streams API and I kept r…
Code reuse at scale: 80 + 80 = 160% ~ phi...coincidence? I think this may become a long horizon harvest for the rigorous OOP strategy, may Bill Joy be disproved. Gray goo may not [taste] like steel-cut oatmeal.
It's often said that π is the factor by which one should multiply all estimates – reducing it to ɸ would be a significant improvement in estimation accuracy!
Re: Measuring the impact of AI on experienced open-source developer productivity
#93It's been very helpful for me. I find ChatGPT the easiest to use; not because it's more accurate (it isn't), but because it seems to understand the intent of my questions most clearly. I don't usually have to iterate much.
I use it like a know-it-all personal assistant that I can ask any question to; even [especially] the embarrassing, "stupid" ones.
> The only stupid question is the one we don't ask.
- On an old art teacher's wall
Re: Measuring the impact of AI on experienced open-source developer productivity
#94Earlier quoted context omitted.
"But I can assure you that "pandas count unique values column 'Foo'" is just as effective an LLM prompt as "Using pandas, how do I get the count of unique values in the column named 'Foo'?"" How can you be so sure? Did you compare in a systematic way or read papers by people who did it? Now I surely get results giving the llm only snippets and keywords, but anything complex, I do notice differences the way I articula…
> How can you be so sure? Did you compare in a systematic way or read papers by people who did it? No, but I didn't need to read scientific papers to figure how to use Google effectively, either. I'm just using a results-based analysis after a lot of LLM usage.
Well, I did needed some tutorials to use google efficently in the old days when + meant something specific.
Re: Measuring the impact of AI on experienced open-source developer productivity
#95Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. You hit the nail on the head here. I feel like I’ve seen a lot of people trying to make strong arguments that AI coding assistants aren’t useful. As someone who uses and enjoys AI coding assistants, I don’t find this research angle to be… uh… very grounded i…
Still odd to me that the only vibe coded software that gets aquired are by companies selling tools or want to promote vibe coding.
Re: Measuring the impact of AI on experienced open-source developer productivity
#96I wonder if the discrepancy is that it felt like it was taking less time because they were having to do less thinking which feels like it is easier and hence faster. Even so... I still would be really surprised if there wasn't some systematic error here skewing the results, like the developers deliberately picked "easy" tasks that they already knew how to do, so implementing them themselves was particularly fast. See…
> which feels like it is easier and hence faster.
We explore this factor in section (C.2.5) - "Trading speed for ease" - in the paper [1]. It's labeled as a factor with an unclear effect, some developers seem to think so, and others don't!
> like the developers deliberately picked "easy" tasks that they already knew how to do
We explore this factor in (C.2.2) - "Unrepresentative task distribution." I think the effect here is unclear; these are certainly real tasks, but they are sampled from the smaller end of tasks developers would work on. I think the relative effect on AI vs. human performance is not super clear...
[1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
Re: Measuring the impact of AI on experienced open-source developer productivity
#97Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. You hit the nail on the head here. I feel like I’ve seen a lot of people trying to make strong arguments that AI coding assistants aren’t useful. As someone who uses and enjoys AI coding assistants, I don’t find this research angle to be… uh… very grounded i…
I think the thing is there IS a learning curve, AND there is a productivity mirage, AND they are immensely useful, AND it is context dependent. All of this leads to a lot of confusion when communicating with people who are having a different experience.
Re: Measuring the impact of AI on experienced open-source developer productivity
#98Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
Well, there are two possible interpretations here of 75% of participants (all of whom had some experience using LLMs) being slower using generative AI:
LLMs have a v. steep and long learning curve as you posit (though note the points from the paper authors in the other reply).
Current LLMs just are not as good as they are sold to be as a programming assistant and people consistently predict and self-report in the wrong direction on how useful they are.
Re: Measuring the impact of AI on experienced open-source developer productivity
#99Earlier quoted context omitted.
This is really disingenuous when you also say that OpenAI and Anthropic have provided you with access and compute credits (on https://metr.org/about ). Not all payment is cash. Compute credits is still by all means compensation.
Are you willing to be compensated with compute credits for your job? Such companies spit out "credits" all over the place in order to gain traction and enstablish themselves. I remember when cloud providers gave vps credits to startups like they were peanuts. To me, it really means absolutelly nothing.
> Are you willing to be compensated with compute credits for your job?
Well, yes? I use compute for some personal projects so I would be absolutely fine if a part of my compensation was in compute credits.
As a company, even more so.
Re: Measuring the impact of AI on experienced open-source developer productivity
#100Earlier quoted context omitted.
> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. You hit the nail on the head here. I feel like I’ve seen a lot of people trying to make strong arguments that AI coding assistants aren’t useful. As someone who uses and enjoys AI coding assistants, I don’t find this research angle to be… uh… very grounded i…
Still odd to me that the only vibe coded software that gets aquired are by companies selling tools or want to promote vibe coding.
That's not odd. These things are incredibly useful and vibe coding mostly sucks.