Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

Post reply on HN
Print view View original

111–120 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#111

Post by acedTrex »

Delk wrote: post #109
acedTrex wrote: post #68

Earlier quoted context omitted.

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though. Bingo, this, so much this. Every dev i know who loves AI stuff was a dev that I had very little technical respect for pre AI. They got some stuff done but there was no craft or quality to it.

For what it's worth, a (former) team mate who was one of the more enthusiastic adopters of gen AI at the time was in fact a pretty good developer who knew his stuff and wrote good code. He was also big on delivering and productivity. In terms of directly generating technical content, I think he mostly used gen AI for more mechanical stuff such as drafting data schemas or class structures, or for converting this or th…

I have certainly used it for different mechanical things. I have copilot, pay for gpt4o etc.

I do think there is a difference between a skilled engineer using it for the mechanical things, and an engineer that OFFLOADS thinking/designing to it.

Theres nuance everywhere, but my original comment was definitely implying the people that attempt to lean on it very hard for their core work.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#112

Post by shadowgovt »

blibble wrote: post #108
shadowgovt wrote: post #87

Earlier quoted context omitted.

The fine art of industry is building more and more elaborate complicated systems atop things someone deeply cares about to be used by those who don't have to deeply care. How many developers do we imagine even know the difference between SIMD and SISD operators, much less whether their software stack knows how to take advantage of SIMD? How many developers do we imagine even know how RAM chips store bits or how a sem…

> How many developers do we imagine even know the difference between SIMD and SISD operators, much less whether their software stack knows how to take advantage of SIMD? How many developers do we imagine even know how RAM chips store bits or how a semiconductor works? hopefully some of those that did a computer science degree? (all of what you've said was part of mine)

Globally, only 41% of software developers even have a bachelor's degree.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#113

Post by Jensson »

randomdata wrote: post #44
null_investor wrote: post #17

They also added lots of technical debt as I'm sure they used the AI to generate tests and some of those tests could be actually testing bugs as the correct behavior. I've already fixed a couple of tests like this, where people clearly used AI and didn't think about it, when in reality it was testing something wrong. Not to mention the rest of the technical debt added... looking at productivity in software development…

> when in reality it was testing something wrong. Must have seen AI write the implementation as well? If you're still cognizant of what you're writing on the implementation side, it's pretty hard to see a test go from failing to passing if the test is buggy. It requires you to independently introduce the same bug the LLM did, which, while not completely impossible, is unlikely. Of course, humans are prone to not unde…

> it's pretty hard to see a test go from failing to passing

Its pretty easy to add a passing test and call it done without checking if it actually fails in the right circumstances, and then you will get a ton of buggy tests.

Most developers don't do the start out at failing and then to passing ritual, especially junior ones who copies code from somewhere instead of knowing what they wrote.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#114

Post by danielvaughn »

jacobsenscott wrote: post #83
moolcool wrote: post #12

Earlier quoted context omitted.

I've found Copilot pretty good at removing some tedium, like it'll write docstrings pretty well most of the time, but it does almost nothing to alleviate the actual mental labour of software engineering

Removing the tedium frees your brain up to focus on the interesting parts of the problem. The "actual mental labor" is the fun part. So I like that.

Yeah this has been my experience. I bucket my work largely into two categories - "creative work" and "toil". I don't want or need AI to replace my creative work, but the more it can handle toil for me, the better.

As it happens, I think co-pilot is a pretty poor user experience, because it's essentially just an autocomplete, which doesn't really help me all that much, and often gets in my way. I like using Cursor with the autocomplete turned off. It gives you the option to highlight a bit of text and either refactor it with a prompt, or ask a question about it in a side chat window. That puts me in the driver seat, so to speak, so I (the user) can reach out to AI when I want to.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#115

Post by choilive »

GoToRO wrote: post #11

For me AI just brought back documentation. All new frameworks lack documentation big time. The last good one for me was a DOS book! I don't think newer developers even have an idea of what good documentation looks like. Even so, AI will propose different things at different times and you still need an experienced developer to make the call. In the end it replaces documentation and typing.

I think AI just made documentation more valuable.

For public facing projects - your documentation just became part of the LLM's training data, so its now extra important your documentation is thorough and accurate because you will have a ton of developers getting answers from that system.

For private projects, your documentation can now be fed into a finetuning dataset or a RAG system, achieving the same effect.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#116

Post by insane_dreamer »

nuancebydefault wrote: post #102
insane_dreamer wrote: post #77

Earlier quoted context omitted.

I haven't found it that useful in my main product development (which while python based, uses our own framework and therefore there's not much code for CoPilot to go on and it usually suggests methods and arguments that don't exist, which just makes extra work). Where I do find it useful are 1) questions about frameworks/languages that I don't work in much and for which there is a lot of example content (i.e., Qt, CS…

It is revolutionary no doubt. How many pre-AI tools would accomplish an amount of 4 big features you mentioned at once?

Well, like I said, a well-designed IDE is a much bigger productivity booster than CoPilot, and I've never heard anyone describe JetBrains, NetBeans or VSCode as "revolutionary".

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#117

Post by danielvaughn »

SoftTalker wrote: post #81
iwontberude wrote: post #64

Earlier quoted context omitted.

Bash is a terrible language in so so many ways and I concur that I have no interest in learning its new features going forward. I remember how painfully inadequate it was for actual computation and its data types are non existent. Gross.

You have to understand the domain in which its intended to be used. Bash scripts are essentially automating what you could do at the command line with utility programs, pipes, redirects, filters, and conditionals. If you're getting very far outside of that scope, bash is probably the wrong tool (though it can be bent to do just about anything if one is determined enough).

I don't know too much about the history, but wasn't this one of the original motivations for Python? Like it was meant to be basically "bash scripting but with better ergonomics", or something like that.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#118

Post by mistrial9 »

simonw wrote: post #110
acedTrex wrote: post #68

Earlier quoted context omitted.

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though. Bingo, this, so much this. Every dev i know who loves AI stuff was a dev that I had very little technical respect for pre AI. They got some stuff done but there was no craft or quality to it.

I’m a developer who loves using AI, and I’m confident that I was a capable developer producing high quality code long before I started using AI tools.

ok but you are not the sort that most of these comments are talking about.. Wasn't your demo at PyCon making jaws drop among hard core techies ?

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#119

Post by godelski »

B(i)ased as fuck.

I think this post from the other day adds some important context[0]. In that study kids with access to GPT did way more practice problems but worse on the test. But the most important part was that they found that while GPT usually got the final answer right that the logic was wrong, meaning that the answer is wrong. This is true for math and code.

There's the joke: there's two types of 10x devs, those that do 10x work and those who finish 10x jira tickets. The problem with this study is the assumptions that it makes, which is quite common and naive in our industry. They assume that PRs and commits are measures of productivity and they assume passing review is a good quality metric. These are so variable between teams. Plenty are just "lgtm" reviews.

The issue here is that there's no real solid metric for things like good code. Meeting the goals of a ticket doesn't mean you haven't solved the problem so poorly you are the reason 10 new tickets will be created. This is the real issue here and the only real way to measure it is using Justice Potter's test (I know it when I see it), and requires an expert evaluator. In other words, tech debt. Which is something we're seeing a growing rise in, all the fucking enshitification.

So I don't think that study here contradicts [0], in fact I think they're aligned. But I suspect people who are poor programmers (or non programmers) will use this at evidence for what they want to see. Believing naive things like lines of code, number of commits/PRs, etc are measures of productivity rather than hints of measure. I'm all for "move fast and break things" as long as there's time set aside to clean up the fucking mess you left behind. But there never is. It's like we have businesses ADHD. There's so much lost productivity because so much focus is placed on short term measurements and thinking. I know medium and long term thinking are hard, but humans do hard shit every day. We can do a lot better than a shoddy study like this.

[0] https://news.ycombinator.com/item?id=41453300

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#120

Post by [deleted] »

jajko wrote: post #101
ang_cire wrote: post #90

I sometimes wonder about whether the decline in IT worker quality is down to companies trying to force more and more roles onto each worker to reduce headcount. Developers, Operations, and Security used to be dedicated roles. Then we made DevOps and some businesses took that to mean they only needed 2/3 of the headcount, rather than integrating those teams. Then we made DevSecOps, and some businesses took that to mea…

So its not going to stop. Typical C-suite who holds real power has absolutely 0 clue about IT complexity, we are overpriced janitors to them. Their fault, their blame, but they are probably long gone when these mistakes manifest fully. In my banking corp, in past 13 years I've seen massive rise of complexity, coupled with absolutely madly done bureaucracy increase. I still could do all stuff that is required but - I…

[deleted]
Post reply on HN