Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

131–140 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#131

A 26% productivity increase sounds inline with in my experience. I think one dimension they should explore is whether you're working with a new technology or one that you're already familiar with. AI helps me much more with languages/frameworks that I'm trying to learn.

I only use (free) ChatGPT sporadically, and it works best for me in areas where I'm familiar enough to call bullshit, but not familiar enough to write things myself quickly / confidently / without checking a lot of docs:

- writing robust bash and using unix/macos tools

- how to do X in github actions

- which API endpoint do I use to do Y

- synthesizing knowledge on some topic that would require dozens of browser tabs

- enumerating things to consider when investigating things. Like "I'm seeing X, what could be the cause, and how I do check if it's that". For example I told it last week "git rebase is very slow, what can it be?" and it told me to use GIT_TRACE=1 which made me find a slow post-commit hook, and suggested how to skip this hook while rebasing.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#132
post #101

Earlier quoted context omitted.

So its not going to stop. Typical C-suite who holds real power has absolutely 0 clue about IT complexity, we are overpriced janitors to them. Their fault, their blame, but they are probably long gone when these mistakes manifest fully. In my banking corp, in past 13 years I've seen massive rise of complexity, coupled with absolutely madly done bureaucracy increase. I still could do all stuff that is required but - I…

- my laundry app (that I must use) takes minutes to load, doesn't cache my last used laundry room, and the list of rooms isn't even fucking sorted (literally: room 3, room 7, room 1, room 2, ...) - my AC's app takes 45 s to load even if I just used it, because it needs to connect. Worse, I'll bring the temp down in my house and in the evening raise it, but it'll come on even when 5F below my target value, staying on…

... it is not "us" .. there is a collection of roles that each interact to produce products to market or releases. Add some game-theory to the thinking on that?

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#133
post #11

For me AI just brought back documentation. All new frameworks lack documentation big time. The last good one for me was a DOS book! I don't think newer developers even have an idea of what good documentation looks like. Even so, AI will propose different things at different times and you still need an experienced developer to make the call. In the end it replaces documentation and typing.

Every now and then there's a HN discussion "how do you manage internal documentation" where most commenters write something like "there's no point writing documentation because it quickly becomes outdated". (There were two such threads in just last days). Might explain why nothing is documented anymore.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#134

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though.

For them programming is a means to an end, and I think it is fine, in a way. But you cannot just ask an AI to write you tiktok clone and expect to get the finished product. Writing software is an iterative process, and LLMs currently used are not good enough for that, because they need not only to answer the questions, but at the very minimum to start asking questions: "why do want to do that ?" "do you prefer this or that", etc., so that they can actually extract all the specification details that the user happily didn't even know he needed before producing an appropriate output. (It's not too different from how some independent developers has to handle their clients, isn't it ?). Probably we will get there, but not too soon.

I also doubt that current tools can keep a project architecturally sound long-term, but that is just an hunch.

I admit though that I may be biased because I don't like much tools like copilot: when I write software, I have in my mind a model of the software that I am writing/I want to write, the AI has another model "in mind" and I need to spend mental energy understanding what it is "thinking". Even if 99/100 it is what I wanted, the remaining 1% is enough to hold me back from trusting it. Maybe I am using it the wrong way, who knows.

The AI tool that work for me would be a "voice controller AI powered pair programmer": I write my code, then from time to time I ask him questions on how to do something, and I can get either an contextual answer depending on the code I am working on, or generate the actual code if I wish so". Are there already plugins working that way for vscode/idea/etc ?

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#135
post #110

Earlier quoted context omitted.

I’m a developer who loves using AI, and I’m confident that I was a capable developer producing high quality code long before I started using AI tools.

ok but you are not the sort that most of these comments are talking about.. Wasn't your demo at PyCon making jaws drop among hard core techies ?

Generalizing the talent and enthusiasm of your peers based on the tools they enjoy to use is going to cause friction.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#136
post #101

I sometimes wonder about whether the decline in IT worker quality is down to companies trying to force more and more roles onto each worker to reduce headcount. Developers, Operations, and Security used to be dedicated roles. Then we made DevOps and some businesses took that to mean they only needed 2/3 of the headcount, rather than integrating those teams. Then we made DevSecOps, and some businesses took that to mea…

So its not going to stop. Typical C-suite who holds real power has absolutely 0 clue about IT complexity, we are overpriced janitors to them. Their fault, their blame, but they are probably long gone when these mistakes manifest fully. In my banking corp, in past 13 years I've seen massive rise of complexity, coupled with absolutely madly done bureaucracy increase. I still could do all stuff that is required but - I…

[deleted]

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#137

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though. Is this a bad thing? Maybe I'm misunderstanding it, but even when I'm working on my own projects, I'm usually trying to solve a problem, and the technology is a means to an end to solving that problem (delivering). I care that it works, and is maintainable, I don't…

Caring that it works and is maintainable is caring about the technology.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#138

My hunch - it's just a hunch - is that LLM-assisted coding is detrimental to one's growth as a developer. I'm fairly certain it can only boost productivity to a certain level - one which may be tedium for more senior developers, but formative for juniors. My experience is that the LLM isn't just used for "boilerplate" code, but rather called into action when a junior developer is faced with a fairly common task they'…

My experience has been that indeed, it is detrimental to juniors. But unlike your take, it is largely a boon to experienced developers. That you suggest "tedium" is involved for more senior developers suggests to me that you haven't given the tooling a fair chance or work with a relatively obscure technology/language.

I think you’ve misunderstood the GP. They are saying AI is useful to seniors for tasks that would otherwise be tedious, but doing those tedious tasks by hand would be formative for juniors, and it is detrimental to their growth when they do them using AI.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#139

Earlier quoted context omitted.

That might be. It might also be a consequence of how flexible people at different stages in their career are. I have seen mostly senior programmers argue why ai tools don't work. Juniors just use them without prejudice.

I find their value depends a lot on what I'm doing. Anything easy I'll get insane leverage, no exaggeration I'll slap together that shit 25x faster. It's seen likely billions of lines of simple CRUD endpoints, so yeah it'll write those flawlessly for you. Anything the difficult or complex, and it's really a coinflip if it's even an advantage, most of the time it's just distracting and giving irrelevant suggestions or…

I think for me, I'm still learning how to make these tools operate effectively. But even only a few months in, it has removed most all the annoying work and lets me concentrate on the stuff that I like. At this point, I'll often give it some context, tell it what to make and it spits out something relatively close. I look it over, call out like 10 things, each time it says "you're right to question..." and we do an iteration. After we're thru that, I tell it to write a comprehensive set of unit tests, it does that, most of them fail, it fixes them, and them we usually have something pretty solid. Once we have that base pattern, I can have it pattern and extend variants after the first solid bit of code. "Using this pattern for style and approach, make one that does XYZ instead."

But what I really appreciate is, I don't have to do the plug and chug stuff. Those patterns are well defined, I'm more than happy to let the LLM do that and concentrate on steering whether it's making a wise conceptual or architectural choice. It really seems to act like a higher abstraction layer. But I think how the engineer uses the tool matters too.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#140

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them though. Bingo, this, so much this. Every dev i know who loves AI stuff was a dev that I had very little technical respect for pre AI. They got some stuff done but there was no craft or quality to it.

SWE has a huge draw because frankly it's not that hard to learn programming, and the bar to clear in order to land a $100-120k work-from-home salary is pretty low. I know more than a few people who career hopped into software engineering after a lackluster non-tech career (that they paid through the nose to get a degree in, but were still making $70k after 5 years). By and large these people seem to just not be "into it", and like you said are more about delivering than actually making good products/services.

However, it does look like LLM's are racing to make these junior devs unnecessary.

Post reply on HN