Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

Post reply on HN
Print view View original

51–60 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#51

Post by fer »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

Disclaimer: I work at a company who sells coding AI (among many other things).

We use it internally and the technical debt is an enormous threat that IMO hasn't been properly gauged.

It's very very useful to carpet bomb code with APIs and patterns you're not familiar with, but it also leads to insane amounts of code duplication and unwieldy boilerplate if you're not careful, because:

1. One of the two big bias of the models is the fact that the training data is StackOverflow-type training data, which are examples and don't take context and constraints into account.

2. The other is the existing codebase, and it tends to copy/repeat things instead of suggesting you to refactor.

The first is mitigated by, well, doing your job and reviewing/editing what the LLM spat out.

The second can only be mitigated once diffs/commit history become part of the training data, and that's a much harder dataset to handle and tag, as some changes are good (refactorings) but other might be not (bugs that get corrected in subsequent commits) and no clear distinction as commit messages are effectively lies (nobody ever writes: bug introduced).

Not only that, merges/rebases/squashes alter/remove/add spurious meanings to the history, making everything blurrier.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#52

Post by SketchySeaBeast »

pqdbr wrote: post #6
danielvaughn wrote: post #2

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

Same impression here. I've been using Cursor for around 10 days on a massive Ruby on Rails project (a stack I've been coding in for +13 years). I didn't enjoy any productivity boost on top of what GitHub Copilot already gave me (which I'd estimate around the 25% mark). However, for crafting a new project from scratch (empty folder) in, say, Node.js, it's uncanny; I can get an API serving requests from a OpenAPI schem…

I feel like this whole "starting a new project" might be the divide between the jaded and excited, which often (but not always) falls between senior and junior lines. I just don't do that anymore. Coding is no longer my passion, it's my profession. I'm working in an established code base that I need to thoughtfully expand and improve. The easy and boilerplate problems are solved. I can't remember the last time I started up a new project, so I never see that side of copilot or cursor. Copilot might at its best when tinkering.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#53

Post by arexxbifs »

My hunch - it's just a hunch - is that LLM-assisted coding is detrimental to one's growth as a developer. I'm fairly certain it can only boost productivity to a certain level - one which may be tedium for more senior developers, but formative for juniors.

My experience is that the LLM isn't just used for "boilerplate" code, but rather called into action when a junior developer is faced with a fairly common task they've still not (fully) understood. The process of experimenting, learning and understanding is then largely replaced by the LLM, and the real skill becomes applying prompt tweaks until it looks like stuff works.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#54

Post by kraftman »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though.

Is this a bad thing? Maybe I'm misunderstanding it, but even when I'm working on my own projects, I'm usually trying to solve a problem, and the technology is a means to an end to solving that problem (delivering). I care that it works, and is maintainable, I don't care that much about the technology.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#55

Post by smokel »

noobermin wrote: post #46

Can someone potentially smarter than me explain how the data, which in table I clearly shows the majority of the means for each experiment metric being less than the SD could even hope to be salvaged? Taken blindly, the results are simply unbelieveable to outright lying, the sort of thing you see submitted to garbage open access journals. The text describing model they employ afterwards is not convincing enough for m…

At least they acknowledge this:

"However, the table also shows that for all outcomes (with the exception of the Build Success Rate), the standard deviation exceeds the pre-treatment mean, and sometimes by a lot. This high variability will limit our power in our experimental regressions below."

What I find even stranger is that the values in the "control" and "treatment" columns are so similar. That would be highly unlikely given the extreme variability, no?

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#56

Post by randomdata »

infecto wrote: post #10
jstummbillig wrote: post #4

Earlier quoted context omitted.

That might be. It might also be a consequence of how flexible people at different stages in their career are. I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.

I wish I could upvote this comment more than once. There does appear to be a prejudice with more senior programmers arguing why it cannot work, how they just cause more trouble, and other various complaints. The tools today are not perfect but they still amaze me at what is being accomplished, even a 10% gain is incredible for something that costs $10/month. I believe progress will be made in the space and the toolin…

> The tools today are not perfect

The trouble is that they seem to be getting worse. Some time ago I was able to write an entire small application by simply providing some guidance around function names and data structures, with an LLM filling in all of the rest of the code. It worked fantastically and really showed how these tools can be a boon.

I want to taste that same thrill again, but these days I'm lucky if I can get something out of it that will even compile, never mind the logical correctness. Maybe I'm just getting worse at using the tools.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#57

Post by randomdata »

empath75 wrote: post #39
danielvaughn wrote: post #2

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

Sounds about right to me, which is why the hysteria about AI wiping out developer jobs was always absurd. _Every_ time there has been a technology that improved developer productivity, developer jobs and pay have _increased_. There is not a limited amount of automation that can be done in the world and the cheaper and easier it gets to automate stuff, the more stuff will be economically viable to automate that wasn't…

The automated elevator is just a tool, but it "wiped out" the elevator operator. Which is really to say not that the elevator operator was wiped out, but that everyone became the elevator operator. Thus, by the transitive properties of supply and demand, the value of operating an elevator declined to nothing.

Said hysteria was built on the same idea. After all, LLMs themselves are just compilers for a programming language that is incredibly similar to spoken language. But as the programming language is incredibly similar to the spoken language that nearly everyone already knows, the idea was that everyone would become the metaphorical elevator operator, "wiping out" programming as a job just as elevator operators were "wiped out" of a job when operating an elevator became accessible to all.

The key difference, and where the hysteria is likely to fall flat, is that when riding in an elevator there isn't much else to do but be the elevator operator. You may as well do it. Your situation would not be meaningfully improved if another person was there to press the button for you. When it comes to programming, though, there is more effort involved. Even when a new programming language makes programming accessible, there remains a significant time commitment to carry out the work. The business people are still best to leave that work to the peons so they can continue to focus on the important things.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#58

Post by danielvaughn »

hobofan wrote: post #24
danielvaughn wrote: post #2

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

I'd also expand it to "languages/frameworks that I'll never properly learn". I'm not great at remembering specific quirks/pitfalls about secondary languages like e.g. what the specific quoting incantations are to write conditionals in Bash, so I rarely wrote bash scripts for automation in the past. Basically only if that was a common enough task to be worth the effort. Same for processing JSON with jq, or parsing wit…

Oh god yes, LLM for bash scripting is fantastic. Bash is one of those languages that I simply don't want to learn. Reading it makes my eyes hurt.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#59

Post by add-sub-mul-div »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

Maintaining bad AI code is the new maintaining bad offshore code.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#60

Post by choilive »

dartos wrote: post #33
sgarland wrote: post #27

Earlier quoted context omitted.

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. Thank you, this says what I have been struggling to describe. The day I lost part of my soul was when I asked a dev if I could give them feedback on a DB schema, they said yes, and then cut me off a few minutes in with, “yeah, I don’t really care [about X].” You don’t care? I’m telling y…

I’ve noticed this more and more since 2020. A lot of people who entered the field in the past 6 or so years are here for the money, obviously. Nothing wrong with that at all, but as someone with a long time programming and technology passion, it’s sad to see that change.

Its been a while since my undergrad (>10 yrs) but many of my peers were majoring in CS or EE/CE because of the money, at the time I thought it was a bit depressing as well.

With a few more years under my belt I realized theres nothing wrong with doing good work and providing yourself/your family a decent living. Not everyone needs the passion in their field to become among the best or become a "10x"er to contribute. We all have different passions, but we all need to pay the bills.

Post reply on HN