Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

Post reply on HN
Print view View original

121–130 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#121

Post by mattgreenrocks »

infecto wrote: post #10
jstummbillig wrote: post #4

Earlier quoted context omitted.

That might be. It might also be a consequence of how flexible people at different stages in their career are. I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.

I wish I could upvote this comment more than once. There does appear to be a prejudice with more senior programmers arguing why it cannot work, how they just cause more trouble, and other various complaints. The tools today are not perfect but they still amaze me at what is being accomplished, even a 10% gain is incredible for something that costs $10/month. I believe progress will be made in the space and the toolin…

The prejudice comes down to whether they want to talk the LLM into the right solution vs applying what they know already. If you know your way around then there’s no need to go through the LLM. I think sr devs often tend to be more task focused so the idea of outsourcing the thinking to an LLM feels like another step to take on.

I find Claude good at helping me find how to do things that I know are possible but I don’t have the right nomenclature for. This is an area where Google fails you, as you’re hoping someone else on the internet used similar terms as you when describing the problem. Once it spits out some sort of jargon I can latch onto, then I can Google and find docs to help. I prefer to use multiple sources vs just LLMs, partially because of hallucination, but also to keep amassing my own personal context. LLMs are excellent as librarians.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#122

Post by okwhateverdude »

nerdjon wrote: post #29
alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

I am curious about this also. I have now seen multiple PR's that I had to review that a method was clearly completely modified by AI with no good reason and when asked why something was changed we just got silence. Not exaggerating, literal silence and then trying to ignore the question and explain the thing we were first asking them to do. Clearly having no idea what is actually in this PR. This was done because we…

> Not saying these tools don't have a place. But people are using it without understanding what it is putting out and not understanding the long term effects it will have on a code base.

It is worse than that. We're all maintaining in our heads the mental sand castle that is the system the code base represents. The abuse of the autocoder erodes that sand castle because the intentions of the changes, which are crucial for mentally updating the sand castle, are not communicated (because they are unknowable). This is same thing with poor commit messages, or poor documentation around requirements/business processes. With enough erosion, plus expected turn over in staff, the sand castle is actually gone.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#123

Post by godelski »

jajko wrote: post #101
ang_cire wrote: post #90

I sometimes wonder about whether the decline in IT worker quality is down to companies trying to force more and more roles onto each worker to reduce headcount. Developers, Operations, and Security used to be dedicated roles. Then we made DevOps and some businesses took that to mean they only needed 2/3 of the headcount, rather than integrating those teams. Then we made DevSecOps, and some businesses took that to mea…

So its not going to stop. Typical C-suite who holds real power has absolutely 0 clue about IT complexity, we are overpriced janitors to them. Their fault, their blame, but they are probably long gone when these mistakes manifest fully. In my banking corp, in past 13 years I've seen massive rise of complexity, coupled with absolutely madly done bureaucracy increase. I still could do all stuff that is required but - I…

- my laundry app (that I must use) takes minutes to load, doesn't cache my last used laundry room, and the list of rooms isn't even fucking sorted (literally: room 3, room 7, room 1, room 2, ...)

- my AC's app takes 45 s to load even if I just used it, because it needs to connect. Worse, I'll bring the temp down in my house and in the evening raise it, but it'll come on even when 5F below my target value, staying on for 15+ minutes leaving us freezing (5F according to __it's thermometer__!)

- my TV controls are slow. Enough that I buffer inputs and wait 2-3 seconds for the commands to play. That pressing the exit button in the same menu (I turn down brightness at night because auto settings don't work, so it's the exact same behavior), idk if I'm exciting to my input, exiting the menu, or just exiting the sub menu. It's inconsistent!

There's so much that I can go on and on and I'm sure you can too. I think one of the worst parts about being a programmer is that I'm pretty sure I know how to solve many of these issues, and in fact sometimes I'll spend days to tear apart the system to actually fix it. Of course to only be undone by updates that are forced (app or whatever won't connect because why is everything done server side ( ┛ ◉ Д ◉ ) ┛ 彡 ┻ ━ ┻ ). Even worse, I'll make PRs on open projects (or open issues another way and submit solutions) that having been working for months and they just go stale while I see other users reporting the same problems and devs working on other things in the same project (I'll even see them deny the behavior or just respond "works for me" closes issue before opener can respond)

I don't know how to stop caring because these things directly affect me and are slowing me down. I mean how fucking hard is it to use sort? It's not even one line!

What the fuck is wrong with us?

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#124

Post by mistrial9 »

shadowgovt wrote: post #87
carlmr wrote: post #84

Earlier quoted context omitted.

>Cloud was a mistake; it’s inculcated people with the idea that chasing efficiency and optimization doesn’t matter, because you can always scale up or out. Similarly Docker is an amazing technology, yet it enabled the dependency tower of babels that we have today. It enabled developers that don't care about cleaning up their depencies. Kubernetes is amazing technology, yet it enabled the developers that don't care to…

The fine art of industry is building more and more elaborate complicated systems atop things someone deeply cares about to be used by those who don't have to deeply care. How many developers do we imagine even know the difference between SIMD and SISD operators, much less whether their software stack knows how to take advantage of SIMD? How many developers do we imagine even know how RAM chips store bits or how a sem…

you insult fine art

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#125

Post by lispisok »

ang_cire wrote: post #90

I sometimes wonder about whether the decline in IT worker quality is down to companies trying to force more and more roles onto each worker to reduce headcount. Developers, Operations, and Security used to be dedicated roles. Then we made DevOps and some businesses took that to mean they only needed 2/3 of the headcount, rather than integrating those teams. Then we made DevSecOps, and some businesses took that to mea…

Like most things the decline in quality is probably multi-faceted. There is also the component where tech became a hot attractive field so people flooded in who only cared about the paycheck and not the craft.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#126

Post by okwhateverdude »

danielvaughn wrote: post #117
SoftTalker wrote: post #81

Earlier quoted context omitted.

You have to understand the domain in which its intended to be used. Bash scripts are essentially automating what you could do at the command line with utility programs, pipes, redirects, filters, and conditionals. If you're getting very far outside of that scope, bash is probably the wrong tool (though it can be bent to do just about anything if one is determined enough).

I don't know too much about the history, but wasn't this one of the original motivations for Python? Like it was meant to be basically "bash scripting but with better ergonomics", or something like that.

You might thinking of Perl. Affectionately known as the swiss army chainsaw of programming languages, it incorporated elements from bash, awk, sed, etc and smushed them together into a language that carried the early Web.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#127

Post by godelski »

alternatex wrote: post #5

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

  > I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts.
It does not

You also may find this post from the other day more illuminating[0], as I believe the actual result strongly hints at what you're guessing. The study is high schoolers doing math. While GPT only has an 8% error rate for the final answer, it gets the steps wrong half the time. And with coding (like math), the steps are the important bits.

But I think people evaluate very poorly when there's ill defined metrics but some metric exists. They over inflate it's value since it's concrete. Like completing a ticket doesn't mean you made progress. Introducing technical debt would mean taking a step back. A step forward in a very specific direction but away from the actual end goal. You're just outsourcing work to a future person and I think we like to pretend this doesn't exist because it's hard to measure.

[0] https://news.ycombinator.com/item?id=41453300

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#128

Post by nuancebydefault »

insane_dreamer wrote: post #116
nuancebydefault wrote: post #102

Earlier quoted context omitted.

It is revolutionary no doubt. How many pre-AI tools would accomplish an amount of 4 big features you mentioned at once?

Well, like I said, a well-designed IDE is a much bigger productivity booster than CoPilot, and I've never heard anyone describe JetBrains, NetBeans or VSCode as "revolutionary".

I would call IDE's "evolutionary". Jetbrains is a better evolutionary version of, say, Eclipse in the day.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#129

Post by throwaway314155 »

arexxbifs wrote: post #53

My hunch - it's just a hunch - is that LLM-assisted coding is detrimental to one's growth as a developer. I'm fairly certain it can only boost productivity to a certain level - one which may be tedium for more senior developers, but formative for juniors. My experience is that the LLM isn't just used for "boilerplate" code, but rather called into action when a junior developer is faced with a fairly common task they'…

My experience has been that indeed, it is detrimental to juniors. But unlike your take, it is largely a boon to experienced developers. That you suggest "tedium" is involved for more senior developers suggests to me that you haven't given the tooling a fair chance or work with a relatively obscure technology/language.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#130

Post by Delk »

It's probably worth going a bit deeper into the paper before picking up conclusions. And I think the study could really do a bit of a better job of summarizing its results.

The abstract and the conclusion only give a single percentage figure (26.08% increase in productivity, which probably has too many decimals) as the result. If you go a bit further, they give figures of 27 to 39 percent for juniors and 8 to 13 percent for seniors.

But if you go deeper, it looks like there's a lot of variation not only by seniority, but also by the company. Beside pull requests, results on the other outcome measures (commits, builds, build success rate) don't seem to be statistically significant at Microsoft, from what I can tell. And the PR increases only seem to be statistically significant for Microsoft, not for Accenture. And even then possibly only for juniors, but I'm not sure I can quite figure out if I've understood that correctly.

Of course the abstract and the conclusion have to summarize. But it really looks like the outcomes vary so much depending on the variables that I'm not sure it makes sense to give a single overall number even as a summary. Especially since statistical significance seems a bit hit-and-miss.

edit: better readability

Post reply on HN