Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

371–380 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#371

Earlier quoted context omitted.

I have been teaching people at my company how to use AI code tools, the learning curve is way worse for developers and I have had to come up with some exercises to try and breakthrough the curve. Some seemingly can’t get it. The short version is that devs want to give instructions instead of ask for what outcome they want. When it doesn’t follow the instructions, they double down by being more precise, the worst thin…

> Interestingly, the best AI assisted devs have often moved to management/solution architecture, and they find the AI code tools brought back some of the love of coding This suggests me though that they are bad at coding, otherwise they would have stayed longer. And I can't find anything in your comment that would corroborate the opposite. So what gives? I am not saying what you say is untrue, but you didn't give any…

I'm not bad at coding. I would say I'm pretty damned good. But coding is a means-to-an-end. I come up with an idea, then I have the long-winded middle bit where I have to write all the code, spin up a DB, create the tables, etc.

LLMs have given me a whole new love of coding, getting rid of the dull grind and letting me write code an order of magnitude quicker than before.

Re: Measuring the impact of AI on experienced open-source developer productivity

#372

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

This is just not true in my experience. Not with the latest models. I routinely manage to 1-shot a whole "thing." e.g. yesterday I needed a Wordpress plugin for a single-time use to clean up a friend's site. I described exactly what I needed, it produced the code, it ran perfect first time and the UI looked like a million dollars. It got me 100% of the way in 0% of the time.

I'm the biggest skeptic, but more and more I'm seeing it get me the bulk of the way with very little back-and-forth. If it was even more heavily integrated in my dev environment, it would save me even more time.

Re: Measuring the impact of AI on experienced open-source developer productivity

#373

I would love to see a comparison of the pull requests generated by each workflow, if possible. My experience with Copilot has generally been that it suggests far more code than I would actually write to solve a specific problem - sometimes adding extra checks where they aren't needed, sometimes just being more verbose than I would be, and oftentimes repeating itself where it would be better to use an abstraction. My…

Check out section AI increasing issue scope (C.2.3) in the paper -- https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf

We speak (the best we can) to changes in amount of code -- I'll note that this metric is quite messy and hard to reason about!

Re: Measuring the impact of AI on experienced open-source developer productivity

#374

Earlier quoted context omitted.

Personally I don't want poor-average developers to be more productive, I want them to be more expert

Sure. But what would you suppose the ratio is between expert, average, and mediocre coders in the average organization? I think a small minority would be in the first category, and I don’t see a technology on the horizon that will change that except for LLMs, which seem like they could make mediocre coders both more productive and produce higher quality output.

They definitely aren't producing higher quality output imo, but definitely producing low quality output faster

That's not a tradeoff that I like

Re: Measuring the impact of AI on experienced open-source developer productivity

#375
post #120

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distortin…

Part of it is that I feel I don't have to put as much mental energy into the coding part. I use my mental energy on the design and ideas, then kinda breeze through the coding now with AI at a much lower mental energy state than I would have when I was typing every single character of every line.

Re: Measuring the impact of AI on experienced open-source developer productivity

#376

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

>It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Apple's Response to iPhone 4 Antenna Problem: You're Holding It Wrong https://www.wired.com/2010/06/iphone-4-holding-it-wrong/

I don't see how the Antennagate can be qualified as "acceptable" since it caused a big public uproar and Apple had to settle a class action lawsuit.

https://www.businessinsider.com/apple-antennagate-scandal-ti...

Re: Measuring the impact of AI on experienced open-source developer productivity

#377

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Maybe, but it isn't hard to think of developer tools where this is the case. This is the entire history of editor and IDE wars. Imagine running this same study design with vim. How well would you expect the not-previously-experienced developers to perform in such a study?

No one is claiming 10x perf gains in vim.

It’s just a fun geeky thing to use with a lot of zany customizations. And after two hellish years of memory muscling enough keyboard bindings to finally be productive, you earned it! It’s a badge of pride!

But we all know you’re still fat fingering ggdG on occasion and silently cursing to yourself.

Re: Measuring the impact of AI on experienced open-source developer productivity

#378

Earlier quoted context omitted.

>It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Apple's Response to iPhone 4 Antenna Problem: You're Holding It Wrong https://www.wired.com/2010/06/iphone-4-holding-it-wrong/

I don't see how the Antennagate can be qualified as "acceptable" since it caused a big public uproar and Apple had to settle a class action lawsuit. https://www.businessinsider.com/apple-antennagate-scandal-ti...

it didnt end the iphone as a brand, or end smart phones altogether though.

how much did that uproar and settlement matter?

Re: Measuring the impact of AI on experienced open-source developer productivity

#379

Earlier quoted context omitted.

Or, one person in 16 has a particular personality, inclined to LLM dependence.

Invoking personality is to the behavioral science as invoking God is to the natural sciences. One can explain anything by appealing to personality, and as such it explains nothing. Psychologists have been trying to make sense of personality for over a century without much success (the best efforts so far have been a five factor model [Big 5] which has ultimately pretty minor predictive value), which is why most behav…

Behaviorism is a relic of the 1950s

Re: Measuring the impact of AI on experienced open-source developer productivity

#380

Earlier quoted context omitted.

> This suggests me though that they are bad at coding, otherwise they would have stayed longer. Or they care about producing value, not just the code, and realized they had more leverage and impact in other roles. > And I can't find anything in your comment that would corroborate the opposite. I didn’t try and corroborate the opposite. Honestly, I don’t care about the “best coders.” I care about people who do their j…

> Honestly, I don’t care about the “best coders.” > Interestingly, the best AI assisted devs have often moved to management/solution architecture Is it just me? Or does it seem to others as well that you pretty much rank these people even at the moment and your first comment contradicts your second comment? Especially when you admit that you rank them based on velocity. I am not saying you shouldn't do that, but it f…

> Is it just me? Or does it seem to others as well that you pretty much rank these people even at the moment and your first comment contradicts your second comment?

I think you are reading what you want to read and not what I said, so yes it is you. The most productive, valuable people with developer titles in my organizations are not the ones who write the cleanest, most beautiful, most perfect code. They do all of the other parts of the job well and write solid code.

Following the introduction of AI tools, many of the people in my organization who most effectively learned to use those tools are people who previously chose to move to manager and SA roles.

Not only are these not contradictory, they fit quite well together. People who do the things around coding well, but maybe have to work hard at writing the actual code, are better at using the AI tools than exceptional coders. For my organization, the former are generally more valuable than the latter without AI, and that is increasing as a result of AI.

> I am not saying you shouldn't do that, but it feels to me like rating road construction workers on the number of potholes fixed, even though it's very possible that the potholes are caused by the sloppy work to begin with.

Not if your measurement includes quality testing the pothole repairs, which mine does, as I explicitly called out. I work in industries with extensive, long testing cycles, we are (imperfectly, of course) able to measure productivity based on things which make it through those cycles.

You are trying very hard to find ways to ignore what I am saying. It is fine if you don’t want to believe me, but these things have been true based on our observations:

A. Great “coders” have a much harder time picking up AI dev tools and using them effectively, and when they see how others use them they will admit that isn’t how they use them. They will revert to their previous habits and give up on the tools.

B. The productivity gains for the people who are good at using the tools, as measured by velocity with a minimum bar for quality (with substantial QA), are very high.

C. We have measured these things to thoroughly understand the ROI and we are accelerating our investment in AI coding tools as a result.

Some caveats I am absolutely willing to make - we are not working on bleeding edge tech doing things no one has ever done before.

We failed to effectively use AI many times before we started to get it right.

There are developers who are slower with the AI code tools than without it.

Post reply on HN