Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the impact of AI on experienced open-source developer productivity

metr.org

Post reply on HN
Print view View original

441–450 of 501 posts

Re: Measuring the impact of AI on experienced open-source developer productivity

#441

Post by [deleted] »

ivanovm wrote: post #309
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

[deleted]

Re: Measuring the impact of AI on experienced open-source developer productivity

#442

Post by curious_cat_163 »

noisy_boy wrote: post #6

It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebble…

100% agreed. It is all about removing friction for me. Case in point: I would not have touched React in my previous career without the assist that LLMs now provide. The barrier to entry just _felt_ to be too large and one always has the instinct to stick with what one knows.

However, it is _fun_ to go over the barrier if it is chatting with a model to get a quick tutorial and produce working code for a prototype (for your specific needs) where the understanding that you just developed is applied. The alternative (without LLMs) is to first do the ground work of learning via tutorials in text/video form and then do the cognitive mapping of applying the learning to one's prototype. I would make a lot of mistakes that expert/intermediate React developers don't make on this path.

One could argue that it shortcuts some learning and perhaps the old way results in better retention. But, our field changes so fast... and when it remains static for too long, projects die. I think of all this as accelerant for progress in adoption of new ways of thinking about software and diffusing that more quickly across the developer population globally. Code is always fungible, anyway. The job is about all the other things that one needs to do besides coding.

Re: Measuring the impact of AI on experienced open-source developer productivity

#443

Post by pdabbadabba »

bluefirebrand wrote: post #374
pdabbadabba wrote: post #333

Earlier quoted context omitted.

Sure. But what would you suppose the ratio is between expert, average, and mediocre coders in the average organization? I think a small minority would be in the first category, and I don’t see a technology on the horizon that will change that except for LLMs, which seem like they could make mediocre coders both more productive and produce higher quality output.

They definitely aren't producing higher quality output imo, but definitely producing low quality output faster That's not a tradeoff that I like

That's the study I'm really interested in: does AI use improve the output of lower-skill developers (not experts). My intuitions point me in the opposite direction. I think AI would improve their work. But I'm not aware of any hard data that would help answer this question.

Re: Measuring the impact of AI on experienced open-source developer productivity

#444

Post by emodendroket »

skydhash wrote: post #422
emodendroket wrote: post #317

Earlier quoted context omitted.

Well, compared to what method that would be faster to answer that kind of question?

Learning the thing. It’s not like I have to use all the libraries of the whole world at the job. You can really fly over a reference documentation if you’re familiar with the domain.

If your job only ever calls on you to use the same handful of libraries of course just becoming deeply familiar is better but that’s obviously not realistic if you’re jumping from this thing to that thing. Nobody would use resources like Stack Overflow either if it were that easy and practical to just “learn the thing.”

Re: Measuring the impact of AI on experienced open-source developer productivity

#445

Post by sarchertech »

biophysboy wrote: post #173
sarchertech wrote: post #129

Earlier quoted context omitted.

That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection. If you are looking for a 0.1% increase in productivity, then 16 is too small.

Well it depends on the variance of the random variable itself. You're right that with big, obvious effects, a larger n is less "necessary". I could see individuals having very different "productivities", especially when the idea is flattened down to completion time.

Individuals do have very different productivities, but they are measuring the productivity difference across a single individual.

Re: Measuring the impact of AI on experienced open-source developer productivity

#446

Post by mwigdahl »

phyzome wrote: post #345
mwigdahl wrote: post #302

Earlier quoted context omitted.

Plus the gaslighting to follow for anyone claiming AI improved their productivity.

Well, it would be a nice counterweight to all the gaslighting of people who claim AI doesn't improve their productivity...

I've never seen a post telling someone who claimed that AI didn't improve their productivity that they were mistaken and that actually it had.

Plenty of "you're holding it wrong" and "it works for me", but not directly telling someone that their perceptions were inaccurate.

Re: Measuring the impact of AI on experienced open-source developer productivity

#447

Post by novaleaf »

ivanovm wrote: post #309
simonw wrote: post #38

Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…

I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…

I've spent the last 2 months trying to figure out how to utilize AI properly, and only in the last week do I feel that I've hit upon a workflow that's actually a force multiplier (vs divisor).

Re: Measuring the impact of AI on experienced open-source developer productivity

#448

Post by Ntrails »

steveklabnik wrote: post #230
Ntrails wrote: post #204

Earlier quoted context omitted.

Some of it is just that (probably different) people said the same damn things 6 months ago. "No, the 2.8 release is the first good one. It massively improves workflows" Then, 6 months later, the study comes out. "Ah man, 2.8 was useless, 3.0 really crossed the threshold on value add" At some point, you roll your eyes and assume it is just snake oil sales

There’s a lot of confounding factors here. For example, you could point to any of these things in the last ~8 months as being significant changes: * the release of agentic workflow tools * the release of MCPs * the release of new models, Claude 4 and Gemini 2.5 in particular * subagents * asynchronous agents All or any of these could have made for a big or small impact. For example, I’m big on agentic tools, skeptica…

> you have to realize you’re talking to a population of people, and not necessarily the same person. Opinions are going to vary, they’re not literally the same person each time.

I pointed this out in my post for a reason. I get it. But even given a different person is saying the same thing every time a new release comes out - the effect on my prior is the same.

Re: Measuring the impact of AI on experienced open-source developer productivity

#449

Post by hajile »

TeMPOraL wrote: post #392
fingerlocks wrote: post #377

Earlier quoted context omitted.

No one is claiming 10x perf gains in vim. It’s just a fun geeky thing to use with a lot of zany customizations. And after two hellish years of memory muscling enough keyboard bindings to finally be productive, you earned it! It’s a badge of pride! But we all know you’re still fat fingering ggdG on occasion and silently cursing to yourself.

> No one is claiming 10x perf gains in vim. Sure they are - or at least were, unitl the last couple years. Same thing with Emacs. It's hard to claim this now, because the entire industry shifted towards webshit and cloud-based practices across the board, and the classical editors just can't keep up with VS Code. Despite the latter introducing LSP, which leveled the playing field wrt. code intelligence itself, the sur…

I use most of the best vim features in VS Code with their vim bindings.

You'd be hard-pressed to find a popular editor without vim bindings.

Re: Measuring the impact of AI on experienced open-source developer productivity

#450

Post by nsingh2 »

blub wrote: post #366
viraptor wrote: post #313

Earlier quoted context omitted.

> It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. It's completely normal in development. How many years of programming experience you need for almost any language? How many days/weeks you need to use debuggers effectively? How long from the first contact with version control until you get git? I think it's the opposite actually - it's common that new classe…

It is completely typical , but at the same time abnormal to have tools with such poor usability. A good debugger is very easy to use. I remember the Visual Studio debugger or the C++ debugger on Windows were a piece of cake 20 years ago, while gdb is still painful today. Java and .NET had excellent integrated debuggers while golang had a crap debugging story for so long that I don’t even use a debugger with it. In fa…

> It is completely typical, but at the same time abnormal to have tools with such poor usability.

The main difference I see is that LLMs are flaky, getting better over time, but still more so than traditional tooling like debuggers.

> Programming languages are notoriously full of unnecessary complexity. Personal pet peeve: Rust lifetime management. If this is what it takes, just use GC (and I am - golang).

Lifetime management is an inherently hard problem, especially if you need to be able to reason about it at compile time. I think there are some arguments to be made about tooling or syntax making reasoning about lifetimes easier, but not trivial. And in certain contexts (e.g., microcontrollers) garbage collectors are out of the question.

Post reply on HN