Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

Post reply on HN
Print view View original

21–30 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#21

Post by anonzzzies »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> had to tackle after the less experienced devs have contributed their AI-driven efforts.

So, like before AI then? I haven't seen AI deliver illogical nonsense that I couldn't even decipher like I have seen some outsourcing companies deliver.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#22

Post by onlyrealcuzzo »

null_investor wrote: post #17

They also added lots of technical debt as I'm sure they used the AI to generate tests and some of those tests could be actually testing bugs as the correct behavior. I've already fixed a couple of tests like this, where people clearly used AI and didn't think about it, when in reality it was testing something wrong. Not to mention the rest of the technical debt added... looking at productivity in software development…

> They also added lots of technical debt as I'm sure they used the AI to generate tests and some of those tests could be actually testing bugs as the correct behavior.

Let's not forget that developers some times do this, too...

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#23

Post by SketchySeaBeast »

jncfhnb wrote: post #13
jstummbillig wrote: post #4

Earlier quoted context omitted.

That might be. It might also be a consequence of how flexible people at different stages in their career are. I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.

My experience is juniors using them without prejudice and then not understanding why their code is wrong

I am seeing that a lot - juniors who can put out a lot of code but when they get stuck they can't unstick themselves, and it's hard for me to unstick them because they have a hard time walking me through what they are doing.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#24

Post by hobofan »

danielvaughn wrote: post #2

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

I'd also expand it to "languages/frameworks that I'll never properly learn".

I'm not great at remembering specific quirks/pitfalls about secondary languages like e.g. what the specific quoting incantations are to write conditionals in Bash, so I rarely wrote bash scripts for automation in the past. Basically only if that was a common enough task to be worth the effort. Same for processing JSON with jq, or parsing with AWK.

Now with LLMs, I'm creating a lot more bash scripts, and it has gotten so easy that I'll do it for process-documentation more often. E.g. what previously was a more static step-by-step README with instructions is now accompanied with an interactive bash script that takes user input.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#25

Post by carlmr »

gamerDude wrote: post #14
pqdbr wrote: post #6

Earlier quoted context omitted.

Same impression here. I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years). I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark). However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schem…

I start new projects a lot and just have a template that has everything I would need already set up for me. Do you think there is a unique value prop that AI gives you when setting up that a template would not have?

AI has templates for the languages you seldom use (as long as the language is not too niche for the LLM to have trained on it).

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#26

Post by lherron »

gamerDude wrote: post #14
pqdbr wrote: post #6

Earlier quoted context omitted.

Same impression here. I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years). I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark). However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schem…

I start new projects a lot and just have a template that has everything I would need already set up for me. Do you think there is a unique value prop that AI gives you when setting up that a template would not have?

I do both. I have a basic template that also includes my specific AI instructions and conventions that are inputs for Aider/Cursor. Best of both worlds.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#27

Post by sgarland »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering.

Thank you, this says what I have been struggling to describe.

The day I lost part of my soul was when I asked a dev if I could give them feedback on a DB schema, they said yes, and then cut me off a few minutes in with, “yeah, I don’t really care [about X].” You don’t care? I’m telling you as the SME for this exactly what can be improved, how to do it, and why you should do so, but you don’t care. Cool.

Cloud was a mistake; it’s inculcated people with the idea that chasing efficiency and optimization doesn’t matter, because you can always scale up or out. I’m not even talking about doing micro-benchmarks (though you should…), I’m talking about dead-simple stuff like “maybe use this data structure instead of that one.”

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#28

Post by htrp »

anonzzzies wrote: post #21
alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> had to tackle after the less experienced devs have contributed their AI-driven efforts. So, like before AI then? I haven't seen AI deliver illogical nonsense that I couldn't even decipher like I have seen some outsourcing companies deliver.

AI seems to give more consistent results than outsourcing (not necessarily better, but at least more predictable failure modes)

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#29

Post by nerdjon »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

I am curious about this also. I have now seen multiple PR's that I had to review that a method was clearly completely modified by AI with no good reason and when asked why something was changed we just got silence. Not exaggerating, literal silence and then trying to ignore the question and explain the thing we were first asking them to do. Clearly having no idea what is actually in this PR.

This was done because we asked for a minor change to be done (talking maybe 5 lines of code) and tested. So now not only are we dealing with new debt, we are dealing with code that no one can explain why it was completely changed (and some of the changes were changes for the sake of change), and we are dealing with those of us that manage this code now looking at completely foreign code.

I keep seeing this with people that are using these tools and they are not higher level engineers. We finally got to the point of denying these PR's and saying to go back and do it again. Loosing any of the time that was theoretically gained from doing it in the first place.

Not saying these tools don't have a place. But people are using it without understanding what it is putting out and not understanding the long term effects it will have on a code base.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#30

Post by dartos »

pqdbr wrote: post #6
danielvaughn wrote: post #2

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

Same impression here. I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years). I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark). However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schem…

How do you use AI in your workflow outside of copilot?

I haven’t been able to get any mileage out of chat AI beyond treating it like a search engine, then verifying what it said…. Which isn’t a speedy workflow

Post reply on HN