Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow data reveals the hidden productivity tax of almost right AI code

venturebeat.com

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#11

Post by Capricorn2481 »

zihotki wrote: post #4

As if humans always produce high quality always right working code without tech debt. The article is full of AI cliche echos.

I've never had a colleague make up a method that doesn't exist and then hand it to me saying it was done. That's not tech debt, it's wrong in a way that's distinct from humans.

The problem with LLMs is they treat all context as significant. Even telling it to ignore previous context is just confusing it more. So Cursor gets hung up on dumb shit like insignificant linter errors or insisting that extracting a block of code to a function will fix a bug (what?). You add more cursor rules to keep it from doing these things, which doesn't focus it, it just pollutes the context more. It doesn't know when to ignore files it's already read. It just gets shittier and shittier.

I appreciate that I can tell someone where the problem is and they'll know what is or isn't irrelevant.

Edit: Here is something Claude Sonnet 4 told me today. I would expect this if I was talking to it for a while, not the second request.

"Used excludeSwitches (plural) instead of excludeSwitches"

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#13

Post by throwawayoldie »

> One of the most surprising findings was a significant shift in developer preferences for AI compared to previous years, while most developers use AI, they like it less and trust it less this year

I bet that is surprising, at least if you're astonishingly naive and believe everything a company tells you when they're trying to sell you something.

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#14

Post by qwertylicious »

Capricorn2481 wrote: post #11
zihotki wrote: post #4

As if humans always produce high quality always right working code without tech debt. The article is full of AI cliche echos.

I've never had a colleague make up a method that doesn't exist and then hand it to me saying it was done. That's not tech debt, it's wrong in a way that's distinct from humans. The problem with LLMs is they treat all context as significant. Even telling it to ignore previous context is just confusing it more. So Cursor gets hung up on dumb shit like insignificant linter errors or insisting that extracting a block of…

Yeah, LLMs share a lot of the same challenges as self-driving cars: when they work great, we get complacent, and then when they fail, they fail in ways that humans are really bad at anticipating.

And the discourse around all this is also the same: the detractors point out these flaws, and the proponents chime in with "Yeah well humans suck too" and around and around we go...

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#15

Post by CJefferson »

belter wrote: post #2

It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. The latest published research has a sample size of like...16 developers? We don't even need human subjects. GitHub contains a natural experiment: Millions of high-quality commits and PRs from before the Ge…

The main problem (and the reason I have no intention of doing any research involving AI programming), is even if you could get the research started, completed, and published in 4 months (and incredibly ambitious goal), 95% of the comments would just be "Oh, but you didn't consider FishGPT 5.2, and Pineapple AI, which came out 6 weeks ago, so any negative points are entirely out of date".

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#16

Post by bigtex »

CJefferson wrote: post #15
belter wrote: post #2

It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. The latest published research has a sample size of like...16 developers? We don't even need human subjects. GitHub contains a natural experiment: Millions of high-quality commits and PRs from before the Ge…

The main problem (and the reason I have no intention of doing any research involving AI programming), is even if you could get the research started, completed, and published in 4 months (and incredibly ambitious goal), 95% of the comments would just be "Oh, but you didn't consider FishGPT 5.2, and Pineapple AI, which came out 6 weeks ago, so any negative points are entirely out of date".

"You didn't use SOTA" which is what I see when people respond to critical statements about the productivity of AI on Twitter and Reddit.

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#17

Post by AIPedant »

bwfan123 wrote: post #3

We are past the "peak of inflated expectations" and entering the "trough of disillusionment" - which are the gartner hype-cycle phases for any new tech [1]. Just for reference (and whether you agree with it or not) the phases are 1) innovation, 2) peak of inflated expectations, 3) trough of disillusionment, 4) enlightenment and finally 5) plateau Of course, stack-overflow has every incentive to push this narrative, w…

The core issue with the Gartner hype cycle (beyond it not being a cycle!) is that it simply does not apply to "any new tech," tautologically it only applies to hype-driven tech with staying power. Tech which is not driven by hype (e.g. mRNA vaccines driven by urgent need) don't have a trough of disappointment, just a steady slope upwards. And tech which is driven entirely by hype (Theranos) stays in the trough of disappointment forever.

The Gartner hype cycle is astrology for MBAs: it cannot fail, it can only be failed by those lacking faith to see its wisdom.

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#18

Post by lelanthran »

belter wrote: post #2

It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. The latest published research has a sample size of like...16 developers? We don't even need human subjects. GitHub contains a natural experiment: Millions of high-quality commits and PRs from before the Ge…

> It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity.

It's much more remarkable, to me, that the AI vendors have been unable to publish a study on the effects of agents on developer velocity.

I mean, if it was really a 10x (or even 2x) boost to productivity, why aren't they seeing it? The returns on each generation have been diminishing compared to the cost. Shouldn't it be the other way around if AI was such a productivity boost?

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#19

Post by AIPedant »

belter wrote: post #2

It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. The latest published research has a sample size of like...16 developers? We don't even need human subjects. GitHub contains a natural experiment: Millions of high-quality commits and PRs from before the Ge…

This has been the case in software development for a very long time, it is almost impossible to get reliable scientific data about these things because it's almost impossible to run a controlled experiment, and even if you did it's very difficult to decide what to measure and how to measure it. (It's easy to say "compare complexity and quality metrics" but how the fk are you going to define that? Does it really make sense to use the same complexity metric for Go and C++?)

Software developers have had many flame wars over the decades because the lack of data forces the conversation to be anecdotal and ideological:

- should we use dynamic/gradual typing and prioritize developer productivity, or static typing to help enforce correctness?

- agile vs waterfall

- OO vs procedural vs functional

- is Rust's fussiness around memory management more trouble than it's worth for large projects?

- when should you use a 3rd-party library vs doing it yourself?

So this really is nothing new. You are badly underestimating the scientific challenges, instead just hoping big data will plow through. It won't.

Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code

#20

Post by chuckadams »

AIPedant wrote: post #17
bwfan123 wrote: post #3

We are past the "peak of inflated expectations" and entering the "trough of disillusionment" - which are the gartner hype-cycle phases for any new tech [1]. Just for reference (and whether you agree with it or not) the phases are 1) innovation, 2) peak of inflated expectations, 3) trough of disillusionment, 4) enlightenment and finally 5) plateau Of course, stack-overflow has every incentive to push this narrative, w…

The core issue with the Gartner hype cycle (beyond it not being a cycle!) is that it simply does not apply to "any new tech," tautologically it only applies to hype-driven tech with staying power. Tech which is not driven by hype (e.g. mRNA vaccines driven by urgent need) don't have a trough of disappointment, just a steady slope upwards. And tech which is driven entirely by hype (Theranos) stays in the trough of dis…

I'm a bit amused to learn that the term "Gartner Hype Cycle" is actually some official term from Gartner. I always thought it was a tongue-in-cheek reference to the fact that Gartner is at is core a hype merchant. Reality laps satire once again.
Post reply on HN