Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Productivity gains from AI coding assistants haven’t budged past 10% – survey

shiftmag.dev

Post reply on HN
Print view View original

61–70 of 99 posts

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#61

Post by MYEUHD »

oblio wrote: post #44
qudat wrote: post #31

Earlier quoted context omitted.

Agreed. The bottleneck is QA/Code review and that is never going away from most corps. I've never worked at a job in tech that didn't require code review and no, asking a code agent to review a PR is never going to be "good enough". And here we are, the central argument for why code agents are not these job killing hype beasts that are so regularly claimed. Has anyone seen what multi-agent code workflows produce? Tak…

> Take a look at openclaw, the code base is an absolute disaster. 500k LoC for something that can be accomplished in 10k. Mission accomplished: acquhire worth probably millions and millions. I agree with you, by the way.

It was a hire not an acquihire. There was no acquisition.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#62

Post by emp17344 »

arctic-true wrote: post #53
onion2k wrote: post #50

A 10% uplift in productivity for the cost of probably 0.001% of the salary budget is an incredible success.

This is exactly right. And assuming organizations use the gains to cut headcount rather than boost total productivity, a 10% reduction in white collar employment would still be an era-defining systemic shock to the economy.

Productivity improvements from automation actually result in an increase in jobs, not fewer jobs. Basic economics.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#63

Post by sshine »

otabdeveloper4 wrote: post #18
vorticalbox wrote: post #9

Earlier quoted context omitted.

one thing that aways slowed me down was writing jsdocs and testing. Now i can write one example of a pass and then get codex to read the code and write a test for all the branches in that section saves time as it can type a lot faster than i can and its mostly copying the example i already have but changing the input to hit all the branches.

> let's have LLMs check our code for correctness Lmao. Rofl even. (Testing is the one thing you would never outsource to AI.)

> Testing is the one thing you would never outsource to AI

That's not really true.

Making the AI write the code, the test, and the review of itself within the same session is YOLO.

There's a ton of scaffolding in testing that can be easily automated.

When I ask the AI to test, I typically provide a lot of equivalence classes.

And the AI still surprises me with finding more.

On the other hand, it's equally excellent at saying "it tested", and when you look at the tests, they can be extremely shallow. Or they can be fairly many unit tests of certain parts of the code, but when you run the whole program, it just breaks.

The most valuable testing when programming with AI (generated by AI, or otherwise) are near-realistic integration tests. That's true for human programmers, but we take for granted that casual use of the program we make as we develop it constitutes as a poor man's test. When people who generally don't write tests start using AI, there's just nothing but fingers crossed.

I'd rather say: If there's one thing you would never outsource to AI, it's final QA.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#64

Post by 8note »

ptx wrote: post #26

Apparently "AI is speeding up the onboarding process", they say. But isn't that because the onboarding process is about learning, and by having an AI regurgitate the answers you can complete the process without learning anything, which might speed it up but completely defeats the purpose?

id guess the time til forst being able to make useful changes has dropped to near zero, but the time to get mastery of the code base has gone towards infinity.

is that mastery still useful as time goes on though? its always felt a bit like its unhealthy for code to have people with mastery on it. its a sign of a bad bus factor. every effort ive ever seen around code quality and documentation improvement has been to make that code mastery and full understanding irrelevant.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#65

Post by emp17344 »

randomtoast wrote: post #35
moralestapia wrote: post #17

AI adoption has reduced productivity at my workplace, and by a noticeable amount!

This will lead to natural selection. As AI becomes increasingly integrated into all areas, companies that manage it less effectively than others will face greater selection pressure.

Or, AI will turn out to just not be that useful.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#66

Post by piva00 »

qudat wrote: post #31
overgard wrote: post #5

You're only as fast as your biggest bottleneck. Adding AI to an existing organization is just going to show you where your bottlenecks are, it's not going to magically make them go away. For most companies, the speed of writing code probably wasn't the bottleneck in the first place.

Agreed. The bottleneck is QA/Code review and that is never going away from most corps. I've never worked at a job in tech that didn't require code review and no, asking a code agent to review a PR is never going to be "good enough". And here we are, the central argument for why code agents are not these job killing hype beasts that are so regularly claimed. Has anyone seen what multi-agent code workflows produce? Tak…

The bottleneck at larger orgs is mostly always decision-making.

Getting code written and reviewed is the trivial part of the job in most cases, discovering the product needs, considering/uncovering edge-cases, defining business logic that is extensible or easily modifiable when conditions change, etc. are the parts that consume 80% of my time.

We in the engineering org at the company I work for have raised this flag many times during adoption of AI-assisting tools, now that the rollout is deeply in progress with most developers using the tools, changing workflows, it has become the sore thumb sticking out: yes, we can deliver more code if it's needed but for what exactly do you need it?

So far I haven't seen a speed up in decision-making, the same chain of approvals, prioritisation, definitions chugs along as it was and it is clearly the bottleneck.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#67

Post by jdlshore »

williamcotton wrote: post #57
jdlshore wrote: post #32

This is self-reported productivity, in that devs are saying AI saves them about 4 hours per week. But let’s not forget the METR study that found a 20% increase in self-reported productivity but a 19% decrease in actual measured productivity. (It used a clever and rigorous technique for measuring productivity differences, BTW, for anyone as skeptical of productivity measures as I am.)

Has the METR study been replicated?

I haven’t heard about any similar studies, no. I’m planning to conduct one at my workplace but we’re still deciding exactly which uses of AI to test.

Re: Productivity gains from AI coding assistants haven’t budged past 10% – survey

#68

Post by kiernanmcgowan »

My biggest road blocks as an engineer has almost never been the authorship of code but everything else around it.

* Getting code reviewed

* Making sure its actually solving the problem

* Communicating to the rest of the team whats happening

* Getting tests to pass

* Getting it deployed

* Verifying that the fix is implemented in production

* Starting it all over when there is a misunderstanding

Slinging more code faster is great and getting unit testing more-or-less for free is awesome but the separation between a good and great engineer is one of communication and management.

AI is causing us to regress to thinking that code velocity is a good metric to use when comparing engineers.

Post reply on HN