Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

381–390 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#381

Post by 8note »

mattmanser wrote: post #276
steveklabnik wrote: post #215

Earlier quoted context omitted.

Yes. In January I would have told you AI tools are bullshit. Today I’m on the $200/month Claude Max plan. As with anything, your miles may vary: I’m not here to tell anyone that thinks they still suck that their experience is invalid, but to me it’s been a pretty big swing.

Ok, I'll have to try it out then. I've got a side project I've 3/4 finished and will let it loose on it. So are you using Claude Code via the max plan, Cursor, or what? I think I'd definitely hit AI news exhaustion and was viewing people raving about this agentic stuff as yet more AI fanbois. I'd just continued using the AI separate as setting up a new IDE seemed like too much work for the fractional gains I'd been s…

id say thats not gonna be the best use for it, unless what you really want is to first document in detail everything about it.

im using claude + vscode's cline extension for the most part, but where it tends to excel is helping you write documentation, and then using that documentation to write reasonable code.

if you're 3/4 of the way done, a lot of the docs of what it wants to work well are gonna be missing, and so a lot of your intentions about why you did or didnt make certain choices will be missing. if you've got good docs, make sure to feed those in as context.

the agentic tool on its own is still kinda meh, if you only try to write code directly from it. definitely better than the non-agentic stuff, but if you start with trying to get it to document stuff, and ask you questions about what it should know in order to make the change its pretty good.

even if you dont get perfect code, or it spins in a feedback loop where its lost the plot, those questions it asks can be super handy in terms of code patterns that you havent thought about that apply to your code, and things that would usually be undefined behaviour.

my raving is that i get to leave behind useful docs in my code packages, and my team members get access to and use those docs, without the usual discoverability problems, and i get those docs for... somewhat slower than i could have written the code myself, but much much faster than if i also had to write those docs

Re: Measuring the impact of AI on experienced open-source developer productivity

#382

Post by davely »

Maxious wrote: post #363
ivanovm wrote: post #309

Earlier quoted context omitted.

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

>It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Apple's Response to iPhone 4 Antenna Problem: You're Holding It Wrong https://www.wired.com/2010/06/iphone-4-holding-it-wrong/

Mobile phone manufacturers were telling users this long before the iPhone was ever invented.

e.g., Nokia 1600 user guide from 2005 (page 16) [0]

[0] https://www.instructionsmanuals.com/sites/default/files/2019...

Re: Measuring the impact of AI on experienced open-source developer productivity

#383

Post by pbasista »

blub wrote: post #366
viraptor wrote: post #313

Earlier quoted context omitted.

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

It is completely typical , but at the same time abnormal to have tools with such poor usability. A good debugger is very easy to use. I remember the Visual Studio debugger or the C++ debugger on Windows were a piece of cake 20 years ago, while gdb is still painful today. Java and .NET had excellent integrated debuggers while golang had a crap debugging story for so long that I don’t even use a debugger with it. In fa…

> git I still have to look up commands for in some cases

I believe that this is okay. One does not need to know the details about every specific git command in order to be able to use it efficiently most of the time.

It is the same with a programming language. Most people are unfamiliar with every peculiarity of every standard library function that the language offers. And that is okay. It does not prevent them from using language efficiently most of the time.

Also in other aspects of life, it is unnecessary to know everything by memory. For example, one does not need to know how to e.g. replace a blade on a lawn mower. But that is okay. It does not prevent them from using it efficiently most of the time.

The point is that if something is done less often, it is unnecessary to remember the specifics of it. It is fine to look it up when needed.

Re: Measuring the impact of AI on experienced open-source developer productivity

#384

Post by 8note »

swayvil wrote: post #49

AI by design can only repeat and recombine past material. Therefore actual invention is out.

its not a huge deal. i dont need the AI to invent a replacement to the for loop or map function; i only want it to use the those tools.

I'm the one providing the invention, it's transforming my invention into an implementation; sometimes better than others.

Re: Measuring the impact of AI on experienced open-source developer productivity

#385

Post by 8note »

zzzeek wrote: post #51

As a project for work, I've been using Claude CLI all week to do as many tasks as possible. So with my week's experience, I'm now an expert in this subject and can weigh in. Two things that stand out to me are 1. it depends a lot on what kind of task you are having the LLM do. and 2. if the LLM process takes more time, it is very likely your cognitive effort was still way less - for sysadmin kinds of tasks, working w…

> esoteric issue or write some subtle documentation where it doesnt really "get" what I'm doing, yeah it is not as productive in that realm and you might want to skip it for the thinking part there

I've been sensing in these cases that i don't have a good enough way to express these problems, and that i actually need to figure that out, or the rest of my team, whether they're using AI or not, are gonna have a real hard time understanding the change i made.

Re: Measuring the impact of AI on experienced open-source developer productivity

#386

Post by tripletao »

AvAn12 wrote: post #225
sarchertech wrote: post #129

Earlier quoted context omitted.

That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection. If you are looking for a 0.1% increase in productivity, then 16 is too small.

“A quarter of the participants saw increased performance, 3/4 saw reduced performance.” So I think any conclusions drawn on these 16 people doesn’t signify much one way or the other. Cool paper but how is this anything other than a null finding?

They show a 95% CI excluding zero in Figure 1. By the usual standards of social science, that's not a null finding. They give their methodology in Appendix D.

For intuition on why it's insufficient to consider N alone, I assume e.g. that you'd greatly increase your belief that a coin was unfair long before 16 consecutive heads--as already noted, the size of the effect also matters. That relationship isn't intuitive in general, and attempts to replace the math with feelings tend to fail.

Re: Measuring the impact of AI on experienced open-source developer productivity

#387

Post by rester324 »

eightysixfour wrote: post #380
rester324 wrote: post #362

Earlier quoted context omitted.

> Honestly, I don’t care about the “best coders.” > Interestingly, the best AI assisted devs have often moved to management/solution architecture Is it just me? Or does it seem to others as well that you pretty much rank these people even at the moment and your first comment contradicts your second comment? Especially when you admit that you rank them based on velocity. I am not saying you shouldn't do that, but it f…

> Is it just me? Or does it seem to others as well that you pretty much rank these people even at the moment and your first comment contradicts your second comment? I think you are reading what you want to read and not what I said, so yes it is you. The most productive, valuable people with developer titles in my organizations are not the ones who write the cleanest, most beautiful, most perfect code. They do all of…

I am not convinced.

If what you write was true, then the rate of bugs of those incredible devs would simply fall to zero at one point, and at that point they would become a legend who we all would have heard of by now. So the whole story sounds too fishy to my taste.

It's OK if you want to manage your team this way. Everyone needs some external feedback to confirm their own bias. It seems you found yours and it works for you.

It's just not a good argument in support of AI or AI assisted development.

It's too anecdotal.

And since you are the one who are telling me that you are right, and not others, it makes me even more skeptical about the whole story.

Re: Measuring the impact of AI on experienced open-source developer productivity

#388

Post by zingar »

blub wrote: post #366
viraptor wrote: post #313

Earlier quoted context omitted.

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

It is completely typical , but at the same time abnormal to have tools with such poor usability. A good debugger is very easy to use. I remember the Visual Studio debugger or the C++ debugger on Windows were a piece of cake 20 years ago, while gdb is still painful today. Java and .NET had excellent integrated debuggers while golang had a crap debugging story for so long that I don’t even use a debugger with it. In fa…

Nitpick: magit for emacs is good enough for everyone whom I’ve seen talk about it describe as “the best git correct” and it is completely free.

Re: Measuring the impact of AI on experienced open-source developer productivity

#389

Post by JelteF »

malfist wrote: post #320
jwrallie wrote: post #303

Earlier quoted context omitted.

You could go the same way as the study, flip a coin to use AI or not, write down the task you just did, the time you thought the task took you and the actual clock time. Repeat and self-evaluate.

Sample size of 16 is already hard enough to draw conclusions from. Sample size of 1 is even worse.

It's the most representative sample size if you're interested in your own performance though. I really don't care if other people are more productive with AI, if I'm the outlier that's not then I'd want to know.

Re: Measuring the impact of AI on experienced open-source developer productivity

#390

Post by nicman23 »

simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

i just treat ai as a very long auto complete. sometimes it surprises me. on things i do not know, like windows C calls, i think i ought to just search the documentation..
Post reply on HN