Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

71–80 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#71
I am an average developer with more than five years experience in Python. I was using chatgpt to create prototypes of what to do in something I was familiar with and was able to debug the task to make it work. I wasn’t specific enough to specify the epaper display had seven colors instead of black and white.

When I was using chatgpt to do qualifiers for a CTF called Hack A Sat at defcon 31 I could not get anything to work such as gnu radio programs.

If you have the ability to debug then I have experienced that it is productive but when you don’t understand you run into problems.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#72
> Notably, less experienced developers showed higher adoption rates and greater productivity gains.

This is what I’ve seen too, I don't think less experienced developers have gotten better in their understanding of anything just more exposed and quicker, while I do think more experience developers have stagnated

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#73

My hunch - it's just a hunch - is that LLM-assisted coding is detrimental to one's growth as a developer. I'm fairly certain it can only boost productivity to a certain level - one which may be tedium for more senior developers, but formative for juniors. My experience is that the LLM isn't just used for "boilerplate" code, but rather called into action when a junior developer is faced with a fairly common task they'…

I was really hoping this study would be exploring that. Really, it's examining short-term productivity gains, ignoring long-term tech debt that can occur, and _completely_ ignoring effects on the growth of the software developers themselves.

I share your hunch, though I would go so far as to call it an informed, strong opinion. I think we're going to pay the price in this industry in a few years, where the pipeline of "clueful junior software developers" is gonna dry way up, replaced by a firehose of "AI-reliant junior software developers", and the distance between those two categories is a GULF. (And of course, it has a knock-on effect on the number of clueful intermediate software developers, and clueful senior software developers, etc...)

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#74

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though. Is this a bad thing? Maybe I'm misunderstanding it, but even when I'm working on my own projects, I'm usually trying to solve a problem, and the technology is a means to an end to solving that problem (delivering). I care that it works, and is maintainable, I don't…

Read it as "closing a ticket" that makes sense. They don't care if the code explodes after the sprint.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#75

> Notably, less experienced developers showed higher adoption rates and greater productivity gains. This is what I’ve seen too, I don't think less experienced developers have gotten better in their understanding of anything just more exposed and quicker, while I do think more experience developers have stagnated

> while I do think more experience developers have stagnated

Is this because they are not using coding assistants? Are they resistant to using them? I have to say that the coding assistant is helpful; it is an ever-present rubber duck that can talk back with useful information.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#77

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

I haven't found it that useful in my main product development (which while python based, uses our own framework and therefore there's not much code for CoPilot to go on and it usually suggests methods and arguments that don't exist, which just makes extra work).

Where I do find it useful are

1) questions about frameworks/languages that I don't work in much and for which there is a lot of example content (i.e., Qt, CSS);

2) very specific questions I would have done a Google Search (usually StackOverflow) for ("what's the most efficient way to CPU and RAM usage on Windows using python") - the result is pointing me to a library or some example rather than directly generating code that I can copy/paste

3) boilerplate code that I already know how to write but saves me a little time and avoids typing errors. I have the CoPilot plugin for PyCharm so I'll write it as a comment in the file and then it'll complete the next few lines. Again best results is something that is very short and specific. With anything longer I almost always have to iterate so much with CoPilot that it's not worth it anymore.

4) a quick way to search documentation

Some people have said it's good at writing unit tests but I have not found that to be the case (at least not the right kind of unit tests).

If I had to quantify it, I'd probably give it a 5-10% increase in productivity. Much less than I get from using a full featured IDE like PyCharm over coding in Notepad, or a really good git client over typing the git commands in the CLI. In other words, it's a productivity tool like many other tools, but I would not say it's "revolutionary".

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#78
post #33

Earlier quoted context omitted.

I’ve noticed this more and more since 2020. A lot of people who entered the field in the past 6 or so years are here for the money, obviously. Nothing wrong with that at all, but as someone with a long time programming and technology passion, it’s sad to see that change.

Its been a while since my undergrad (>10 yrs) but many of my peers were majoring in CS or EE/CE because of the money, at the time I thought it was a bit depressing as well. With a few more years under my belt I realized theres nothing wrong with doing good work and providing yourself/your family a decent living. Not everyone needs the passion in their field to become among the best or become a "10x"er to contribute.…

Yeah, I think that the quality of work (skill + conscientiousness), and the motivation for doing the work, are two separate things.

Off-the-cuff, three groups:

1. There are people who are motivated by having a solid income, yet they take the professionalism seriously, and do skilled rock-solid work, 9-5. I'd be happy to work with these people.

2. There are other people who are motivated by having a solid or more-than-solid income, and (regardless of skill level), it's non-stop sprint performance art, gaming promotion metrics, resume-driven development, practicing Leetcode, and hopping at the next opportunity regardless of where that leaves the project and team.

3. Then there's those weirdos who are motivated by something about the work itself, and would be doing it even if it didn't pay well. Over the years, these people spend so much time and energy on the something, that they tend to develop more and stronger skills than the others. I'd be happy to work with these people, so long as they can also be professional (including rolling up sleeves for the non-fun parts), or amenable to learning to be professional.

Half-joke: The potential of group #3 is threatening to sharp-elbowed group #2, so group #2 neutralizes them via frat gatekeeping tactics (yeah-but-what-school-did-you-go-to snobbery, Leetcode shibboleth for nothing but whether you rehearsed Leetcode rituals, cliques, culture fit, etc.).

Startups might do well to have a mix of #3 and #1, and to stay far away from #2. But startups -- especially the last decade-plus of too many growth investment scams -- are often run by affluent people who grew up being taught #2 skills (for how you game your way into prestigious school, aggressively self-interested networking and promoting yourself, etc.).

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#79
post #33

Earlier quoted context omitted.

I’ve noticed this more and more since 2020. A lot of people who entered the field in the past 6 or so years are here for the money, obviously. Nothing wrong with that at all, but as someone with a long time programming and technology passion, it’s sad to see that change.

There’s certainly nothing wrong with enjoying the high pay, no – I definitely do. But yeah, it’s upsetting to find out how few people care. Even moreso when they double down and say that you shouldn’t care either, because it’s been abstracted away, blah blah blah. Who do you think is going to continue creating these abstractions for you?

I get the pragmatism argument, but I would like to think certain professions should hold themselves to a higher standard. Doctors, lawyers, and engineers have a duty to society IMO that runs counter to a “just mail it in to cash a paycheck” mentality. I guess it comes down to whether you consider software developers to be that same kind of engineer. Certainly I don’t want safety critical software engineers to have that cavalier attitude (although I’ve seen it).

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#80

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though.

You get what you measure. Nobody measure software quality.

Post reply on HN