Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

11–20 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#11
For me AI just brought back documentation. All new frameworks lack documentation big time. The last good one for me was a DOS book! I don't think newer developers even have an idea of what good documentation looks like.

Even so, AI will propose different things at different times and you still need an experienced developer to make the call. In the end it replaces documentation and typing.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#12

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

I've found Copilot pretty good at removing some tedium, like it'll write docstrings pretty well most of the time, but it does almost nothing to alleviate the actual mental labour of software engineering

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#13

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

That might be. It might also be a consequence of how flexible people at different stages in their career are. I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.

My experience is juniors using them without prejudice and then not understanding why their code is wrong

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#14
post #6

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

Same impression here. I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years). I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark). However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schem…

I start new projects a lot and just have a template that has everything I would need already set up for me. Do you think there is a unique value prop that AI gives you when setting up that a template would not have?

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#15

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

That might be. It might also be a consequence of how flexible people at different stages in their career are. I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.

As a senior, you know the problem is actually finishing a project. That's the moment all those bad decisions made by junior need to be fixed. This also means that a 80% done project is more like 20% done, because in the state it is, it can not be finished: you fix one thing and break 2 more.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#16
post #6

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

Same impression here. I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years). I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark). However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schem…

To fully extract maximum value out of LLMs, you need to change how you architect and organize software to make it easy for them to work with. LLMs are great with function libraries and domain specific languages, so the more of your code you can factor into those sorts of things the greater a speed boost they'll give you.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#17
They also added lots of technical debt as I'm sure they used the AI to generate tests and some of those tests could be actually testing bugs as the correct behavior.

I've already fixed a couple of tests like this, where people clearly used AI and didn't think about it, when in reality it was testing something wrong.

Not to mention the rest of the technical debt added... looking at productivity in software development by amount of tasks is so wrong.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#18

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

That might be. It might also be a consequence of how flexible people at different stages in their career are. I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.

As a senior, I find that trying to use copilot really only gives me gains maybe half the time, the other half the time it leads me in the wrong direction. Googling tends to give me a better result because I can actually move through the data quicker. My belief is this is because when I need help I'm doing something uncommon or hard, as opposed to juniors who need help doing regular stuff which will have plenty of examples in the training data. I don't need help with that.

It certainly has its uses - it's awesome at mocking and filling in the boilerplate unit tests.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#19
This is a decently-thorough study, using PRs as a productivity metric while also tracking build failures (which remained constant at MSFT but increased at Accenture).

Would love to see it replicated by researchers at a company that does not have a clear financial interest in the outcome (the corresponding author here was working at Microsoft Research during the study period).

> Before moving on, we discuss an additional experiment run at Accenture that was abandoned due to a large layoff affecting 42% of participants

Eek

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#20

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

[deleted]
Post reply on HN