Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

Post reply on HN
Print view View original

41–50 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#41

Post by sgarland »

hobofan wrote: post #24
danielvaughn wrote: post #2

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

I'd also expand it to "languages/frameworks that I'll never properly learn". I'm not great at remembering specific quirks/pitfalls about secondary languages like e.g. what the specific quoting incantations are to write conditionals in Bash, so I rarely wrote bash scripts for automation in the past. Basically only if that was a common enough task to be worth the effort. Same for processing JSON with jq, or parsing wit…

While I’ll grant you that LLMs are in fact shockingly good with shell scripts, I also highly recommend shellcheck [0]. You can get it as a treesitter plugin for nvim, and I assume others, so it lints as you write.

[0]: https://www.shellcheck.net/

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#42

Post by heisenberg1 »

This is obvious. Right, of course you get an increase in productivity, especially as a junior - when current AI is able to solve leetcode.

BUT I think a lot of people mentioned that, you get code - that the person which wrote it do not understand. So the next time you get a bug there, good luck fixing it.

My take so far. AI is great, but only for non critical, non core code. Everything that is done for plotting and scripting is awesome (which can take days to implement and in minutes with AI) - but core lib functions - wouldn't outsource it to the AI right now.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#44

Post by randomdata »

null_investor wrote: post #17

They also added lots of technical debt as I'm sure they used the AI to generate tests and some of those tests could be actually testing bugs as the correct behavior. I've already fixed a couple of tests like this, where people clearly used AI and didn't think about it, when in reality it was testing something wrong. Not to mention the rest of the technical debt added... looking at productivity in software development…

> when in reality it was testing something wrong.

Must have seen AI write the implementation as well?

If you're still cognizant of what you're writing on the implementation side, it's pretty hard to see a test go from failing to passing if the test is buggy. It requires you to independently introduce the same bug the LLM did, which, while not completely impossible, is unlikely.

Of course, humans are prone to not understanding the requirements, and introducing what isn't really a bug in the strictest sense but rather a misfeature.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#45

Post by SketchySeaBeast »

carlmr wrote: post #31
SketchySeaBeast wrote: post #23

Earlier quoted context omitted.

I am seeing that a lot - juniors who can put out a lot of code but when they get stuck they can't unstick themselves, and it's hard for me to unstick them because they have a hard time walking me through what they are doing.

I've gotten responses now on PRs of the form. "I don't know either, this is what Copilot told me." If you don't even understand your own PR, I'm not sure why you expect other people can. I have used LLMs myself, but mostly for boilerplate and one-off stuff. I think it can be quite helpful. But as soon as you stop understanding the code it generates you will create subtle bugs everywhere that will cost you dearly in t…

Copilot is best for me when it's letting me hit tab to auto-complete something I was going to write anyways.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#46

Post by noobermin »

Can someone potentially smarter than me explain how the data, which in table I clearly shows the majority of the means for each experiment metric being less than the SD could even hope to be salvaged? Taken blindly, the results are simply unbelieveable to outright lying, the sort of thing you see submitted to garbage open access journals. The text describing model they employ afterwards is not convincing enough for me and seems light on details. I mean, wouldn't any reasonable reviewer demand more?

I know preprints don't need polish but this is even below the standard of a preprint, imo.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#47

Post by sgarland »

dartos wrote: post #33
sgarland wrote: post #27

Earlier quoted context omitted.

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. Thank you, this says what I have been struggling to describe. The day I lost part of my soul was when I asked a dev if I could give them feedback on a DB schema, they said yes, and then cut me off a few minutes in with, “yeah, I don’t really care [about X].” You don’t care? I’m telling y…

I’ve noticed this more and more since 2020. A lot of people who entered the field in the past 6 or so years are here for the money, obviously. Nothing wrong with that at all, but as someone with a long time programming and technology passion, it’s sad to see that change.

There’s certainly nothing wrong with enjoying the high pay, no – I definitely do. But yeah, it’s upsetting to find out how few people care. Even moreso when they double down and say that you shouldn’t care either, because it’s been abstracted away, blah blah blah. Who do you think is going to continue creating these abstractions for you?

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#48

Post by noobermin »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

It doesn't, it looks at PRs, commits, and builds only. They remark at this lacking context.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#49

Post by smokel »

Interestingly, they compare number of pull requests as a statistic for productivity. Not that I know of a better metric, but I wonder if this is an accurate metric. It seems similarly misguided as looking at lines of code.

If an AI tool makes me more productive, I would probably either spend the time won browsing the internet, or use it to attempt different approaches to solve the problem at hand. In the latter case, I would perhaps make more reliable or more flexible software. Which would also be almost impossible to measure in a scientific investigation.

In my experience, the differences in developer productivity are so enormous (depending on existing domain knowledge, motivation, or management approach), that it seems pretty hard to make any scientific claim based on looking at large groups of developers. For now, I prefer the individual success story.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#50

Post by ergonaught »

The result is "less experienced people got more stuff done". I do not see an assessment of whether the stuff that got done was well done.

The output of these tools today is unsafe to use unless you possess the ability to assess its correctness. The less able you are to perform that assessment, the more likely you are to use these tools.

Only one of many problems with this direction, but gravity sucks, doesn't it.

Post reply on HN