Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

Post reply on HN
Print view View original

91–100 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#91

Post by prophesi »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

From my experience, most of it is quickly caught in code review. And after a while it occurs less and less, granted that the junior developer puts in the effort to learn why their PRs aren't getting approved.

So, pretty similar to how it was before. Except that motivated junior developers will improve incredibly fast. But that's also kind of always been the case in software development these past two decades?

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#92

Post by yunwal »

pqdbr wrote: post #6
danielvaughn wrote: post #2

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

Same impression here. I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years). I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark). However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schem…

If I struggle on a particularly hard implementation detail in a large project, often I'll use an LLM to set up a boilerplate version of the project from scratch with fewer complications so that I can figure out the problem. It gets confused if it's interacting with too many things, but the solutions it finds in a simple example can often be instructive.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#93

Post by slt2021 »

einpoklum wrote: post #63

Reminds me of a situation I've been in a few times already: Dev: Hey einpoklum, how do I do XYZ? Me: Hmm, I think I remember that... you could try AB and then C. Dev: Ok, but isn't there a better/easier way? Let me ask ChatGPT. ... Dev: Hey einpoklum, ChatGPT said I should do AB and then C. Me: Let me have a look at that for a second. Me: -Right, so it's just what I read on StackOverflow about this, a couple of years…

true, ChatGPT even imports libraries and configs that are 4-5 year outdated.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#94

Post by Bjorkbat »

simpleranchero wrote: post #76

Microsoft people in research team proving Microsoft tools are good. Elsevier will now do ads in research papers.

This is far, far more rigorous than the experiment Microsoft behind Microsoft's claim that Copilot made devs 55% faster.

The experiment in question was to split 95 devs into two groups and see how long it took each group to setup a web server in Javascript. Control took a little under 3 hours on average, the copilot group took 1 hour and 11 minutes on average.

https://github.blog/news-insights/research/research-quantify...

And it is thanks to this weak experiment that Github proudly boasts that Copilot makes devs 55% faster.

By contrast the conclusion that Copilot makes devs ~25% more productive seems reasonable, especially when you read the actual paper and find out that among senior devs the productivity gains are more marginal.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#95

Post by rm_-rf_slash »

danielvaughn wrote: post #2

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

AI speeds my coding up 2x-4x, depending on the breadth of requirements and complexity of new tech to implement.

But coding is just a fraction of my weekly workload, and AI has been less impactful for other aspects of project management.

So overall it’s 25%-50% increase in productivity.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#96

Post by yieldcrv »

dpflan wrote: post #75
yieldcrv wrote: post #72

> Notably, less experienced developers showed higher adoption rates and greater productivity gains. This is what I’ve seen too, I don't think less experienced developers have gotten better in their understanding of anything just more exposed and quicker, while I do think more experience developers have stagnated

> while I do think more experience developers have stagnated Is this because they are not using coding assistants? Are they resistant to using them? I have to say that the coding assistant is helpful; it is an ever-present rubber duck that can talk back with useful information.

Across teams and organizations what I’ve seen this year is that the “best” developers haven’t looked. The proverbial rockstars with the most domain knowledge have never gotten around to AI and LLMs at all.

This is compounded by adherence to misguided corporate policies that broadly prohibit use of LLMs but are meant to only be about putting trade secrets into the cloud, not distinguishing between cloud vs locally run language models. Comfortable people would never challenge this policy with critical thinking, and it requires special interest to look at locally run language models, even just to choose which one to run.

Many developers have not advocated for more RAM on their company issued laptop to run better LLMs.

and I haven't seen any internal language model that the company is running in their intranet. But it would be cool if there was a huggingface-style catalogue and server farm companies could have and let their employees choose models to prompt, always having the latest models to load.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#97

Post by wahnfrieden »

acedTrex wrote: post #68
alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though. Bingo, this, so much this. Every dev i know who loves AI stuff was a dev that I had very little technical respect for pre AI. They got some stuff done but there was no craft or quality to it.

You're playing status games

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#98

Post by YeGoblynQueenne »

prophesi wrote: post #91
alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

From my experience, most of it is quickly caught in code review. And after a while it occurs less and less, granted that the junior developer puts in the effort to learn why their PRs aren't getting approved. So, pretty similar to how it was before. Except that motivated junior developers will improve incredibly fast. But that's also kind of always been the case in software development these past two decades?

It's interesting to see your comment currently right next to that of nerdjon here:

https://news.ycombinator.com/item?id=41465827

(the two comments might have moved apart by the time you read this).

Edit: yep, they just did.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#99

Post by yifanl »

ang_cire wrote: post #90

I sometimes wonder about whether the decline in IT worker quality is down to companies trying to force more and more roles onto each worker to reduce headcount. Developers, Operations, and Security used to be dedicated roles. Then we made DevOps and some businesses took that to mean they only needed 2/3 of the headcount, rather than integrating those teams. Then we made DevSecOps, and some businesses took that to mea…

They're cutting headcount because they have no conception of how to make a good product, so reducing costs is the only of making the bottom line go up.

That's across the board, from the startups whose business plan is to be acquired at all costs, to the giant tech companies, whose business plan is to get monopoly power first, then figure out how to extract money later.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#100

Post by wahnfrieden »

jacobsenscott wrote: post #80
alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though. You get what you measure. Nobody measure software quality.

Maybe not at your workplaces, but at mine, we measured bugs, change failure rate, uptime, "critical functionality" uptime, regressions, performance, CSAT, etc. in addition to qualitative research on quality in-team and with customers
Post reply on HN