Live data from Hacker News

Effects of Gen AI on High Skilled Work: Experiments with Software Developers

papers.ssrn.com

321–330 of 532 posts

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#321
post #298
post #130

It's probably worth going a bit deeper into the paper before picking up conclusions. And I think the study could really do a bit of a better job of summarizing its results. The abstract and the conclusion only give a single percentage figure (26.08% increase in productivity, which probably has too many decimals) as the result. If you go a bit further, they give figures of 27 to 39 percent for juniors and 8 to 13 perc…

To get a better picture of how this comes about: Microsoft has a study for their own internal product use, and wants to show its efficacy. The results aren’t as broadly successful as one would hope. Accenture is the kind of company that cooperates and co-markets with large orgs like Microsoft. With ~300 devs in the pool they hardly move the population at all, and they cannot be assumed to be objective since they are…

Yeah, it does seem fishy.

The third company seems a bit weird to include in other ways as well. In raw numbers in table 1, there seem to be exactly zero effects from the use of CoPilot. Through the use of their regression model -- which introduces other predictors such as developer-fixed and week-fixed effects -- they somehow get an estimated effect of +54%(!) from CoPilot in the number of PRs. But the standard deviations are so far through the roof that the +54% is statistically insignificant within the population of 3000 devs.

Also, they explain the introduction of the week fixed effect as a means of controlling for holidays etc., but to me it sounds like it could also introduce a lot of unwarranted flexibility into the model. But this is a part where I don't understand their methodology well enough to tell whether that's a problem or not.

I generally err towards the benefit of the doubt when I don't fully understand or know something, which is why I focused more on the presentation of the results than on criticizing the study and its methodology in general. I'd have been okay with the summary saying "we got an increase of 27.3% for Microsoft and no statistically significant results for other participants".

But perhaps I should have been more critical.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#322

I wonder if the study includes the technical debt that more experienced developers had to tackle after the less experienced devs have contributed their AI-driven efforts. Because my personal experience has involved a lot of that in one of the companies listed in the study. Also, I've personally seen more interest in AI in devs that have little interest in technology, but a big interest in delivering. PMs love them th…

> had to tackle after the less experienced devs have contributed their AI-driven efforts. So, like before AI then? I haven't seen AI deliver illogical nonsense that I couldn't even decipher like I have seen some outsourcing companies deliver.

> I haven't seen AI deliver illogical nonsense

I have. If you're doing niche-er stuff it doesn't have enough data and hallucinates. The worst is when it spits two screens of code instead of 'this cannot be done at the level you want it'.

> that I couldn't even decipher

That's unrelated to code quality. Especially with C++ which has become as write only as perl.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#323

Earlier quoted context omitted.

> that I couldn't even decipher This is one of the challenges of being a tech lead. Sometimes code is hard to comprehend. In my experience, AI delivered code is no worse than entry level developer code.

But that is a HN bubble thing: I work and worked with seniors with 10-15 years under their belt who have no logic bone in their body; the worst (and an AI does not do that) is when there is a somewhat 'busy' function that has an if or switch statement and, over time, to add features or fix bugs, ifs were added. Now after 5+ years, this function is 15000 lines and is somewhat of a trained neural network adjacent thing…

> to add features or fix bugs, ifs were added.

"Back to that two page function. Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. "

https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Of course, you can try to split the 15000 line function in logical blocks or something, but don't assume the ifs in it are useless.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#324

I sometimes wonder about whether the decline in IT worker quality is down to companies trying to force more and more roles onto each worker to reduce headcount. Developers, Operations, and Security used to be dedicated roles. Then we made DevOps and some businesses took that to mean they only needed 2/3 of the headcount, rather than integrating those teams. Then we made DevSecOps, and some businesses took that to mea…

This is happening across the board, not just to IT workers, and I suspect it's the major factor as for why the expected productivity improvements from technology didn't materialize. Think of your "DevSecOps" people doing 3x the work they should. Do you know what else are they doing? Booking their own business travels. Reconciling and reporting their own expenses. Reporting their own hours, broken down by business cat…

And yet we are told that competition will determine that capital and talent will flow towards that most efficient organisations over time. Thus, surely organisations that eschewed this practice would emerge and dominate?

So, either capitalism doesn't work, or your thesis isn't quite right...

I have two other counters to offer, first we have seen GDP per capita gradually increasing in major economies for the last 50 years (while the IT revolution has played out). There have been other technical innovations over this time, but I believe that GDP per capita has more than quadrupled in G8 economies. The USA and Canada have, at the same time, enjoyed a clear extra boost from fracking and shale extraction, and the USA has arguably enjoyed an extra extra boost from world dominance - but arguably.

The second one is simple anecdote. Hour for hour I now can do far more in terms of development than I did when I was a hard core techie in the 90's and 2000's. In addition I can manage and administer systems that are far more complex than those it took teams of people to run at that time (try running a 10gb size db under load on oracle 7 sitting on top of spinning rust and 64mb ram store for fun) I can also manage a team of 30's expenses, timesheets, travel requests and so on that again would have taken a person to do. I can just do these things and my job as well and I do it mostly in about 50 hrs a week. If I wasn't involved in my people's lives and happy to argue with customers to get things better I could do it in 40 hrs regularally, for sure. But I put some discretion in.

My point is - we are just more productive. It is hard to measure, and anecdote / "lived experience" is a bad type of evidence, but I think it's clearly there. This is why then accountants have been able to reorganise modern business organisations to use fewer people to do more. Have they destroyed value while doing this - totally - but they have managed to get away with it because 7/10 they have been right.

Personally I've suffered from the 3/10 errors. I know many of us on here have, but we shouldn't shut our eyes because of that.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#325

I sometimes wonder about whether the decline in IT worker quality is down to companies trying to force more and more roles onto each worker to reduce headcount. Developers, Operations, and Security used to be dedicated roles. Then we made DevOps and some businesses took that to mean they only needed 2/3 of the headcount, rather than integrating those teams. Then we made DevSecOps, and some businesses took that to mea…

One person can do the work of 3 and regularly does in startups. I think that what the MBAs miss is this phenomena of overconstraint . Once you have separate the generic role of "developer" into "developer, operations, and security", you've likely specified all sorts of details about how those roles need to be done. When you combine them back into DevSecOps, all the details remain, and you have one person doing 3x the…

I think it less to do with judgement and more to do with one person can do the work of 3 in startups because there's several orders of magnitude less coordination and communication that needs to happen.

If you have 5 people in a startup you have 10 connections between them, 20 people = 190 connections, 100 = 4950 connections, 1000 people = 499500 connections.

Sure you split then in groups with managers and managers managers etc to break down the connections to less than the max but it's still going to be orders of magnitude more communication and coordination needed than in a startup.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#326

Earlier quoted context omitted.

One person can do the work of 3 and regularly does in startups. I think that what the MBAs miss is this phenomena of overconstraint . Once you have separate the generic role of "developer" into "developer, operations, and security", you've likely specified all sorts of details about how those roles need to be done. When you combine them back into DevSecOps, all the details remain, and you have one person doing 3x the…

I think there is another bit to this which is cargo cult tendencies. Basically DevOps is a great idea under certain circumstances and works well for specific people in that role. Realistically if you take a top talent engineer they can likely step into any of the three roles or even some others and be successful. Having the synergy of one person being responsible for the boundary between two roles then makes your ROI…

> There is no methodology (organizational, programming, testing, etc.) that can make excellence out of mediocrity.

This is a thought provoking phrase, and I liked that. Thinking a bit deeper, I'm not sure if it's accurate, practical or healthy though.

I've seen mediocrity produce excellent solutions, and excellence churn out rubbish, and I suspect most people with a few years and tech jobs under their belt have.

You could argue that if they turned out excellent things, then by definition, they're excellent, but that then makes the phrase uselessly tautological.

If it's true, then what's the advice to mediocre people and environments - just give up? Don't waste your time on excellent organisation, programming, testing, because it's a waste and you'll fail?

I think there's no 1 thing that can make excellence out of mediocrity for everyone. But I like to think that for every combination of work, people and environment, there's an optimal set of practices that will lead to them producing something better than average. Obviously a lot don't find that set of practices due to the massive search space.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#327
post #232

Earlier quoted context omitted.

The assumption is that they almost certainly did not have the sample size to justify two decimal places. If they want two decimals for aesthetics over properly honoring significant figures then it calls the scientific rigor into question.

But they say "26.08% increase (SE: 10.3%)", so they make it clear that there's a lot of uncertainty around that number. They could have said "26% (rounded to 0 dp)" or something, but that conveys even less information about the amount of uncertainty than just saying what the standard error is.

They could still have gone for one decimal. Or possibly even none, considering the magnitude of the SE, but I get that they might not want to say "SE: 10%".

The second decimal point doesn't essentially add any information because the data can't provide information at that precision. It's noise but being included in the summary result makes it implicitly look like information. Which is exactly why including it seems a bit questionable.

That's not the major issue with the study, though, it's just one of the things that caught my eye originally.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#328
post #318
post #226

Earlier quoted context omitted.

I would also like to see those numbers. IMHO, the biggest difference comes from housing cost. For example, from my own back-of-the-envelope calculations, a £100k Oxbridge job affords a better lifestyle than a $180k NY job mostly because of housing. However, some US areas have competitively priced housing and jobs that would make the balance tilt in favor of America. In EU, affordable spots with lots of desirable loca…

I don't know how NY is in comparison, but housing in Cambridge is almost as expensive as in London. A detached 3br starts at £700k. NIMBYs keep killing any expansion of supply.

Some say NIMBY's other say water supply.

"Water supply issues are already holding back housing development around the city. In its previous draft water resources management plan, Cambridge Water failed to demonstrate that there was enough to supply all of the new properties in the emerging local plan without risk of deterioration.

The Environment Agency recently confirmed that it had formally objected to five large housing developments in the south of the county because of fears they could not sustainably supply water. It has warned that planning permission for more than 10,000 homes in the Greater Cambridge area and 300,000 square metres of research space at Cambridge University are in jeopardy if solutions cannot be found.

A document published alongside the Case for Cambridge outlines the government’s plan for a two-pronged approach to solving the water scarcity issue, to be led by Dr Paul Leinster, former chief of the Environment Agency, who will chair the Water Scarcity Group.

In the long term, supply will be increased, initially through two new pieces of infrastructure: a new reservoir in the Fens will delivery 43.5 megalitres per day, while a new pipeline will transfer 26 megalitres per day from Grafham Water, currently used by Affinity Water.

But, according to Kelly, a new reservoir would only solve supply requirements for the existing local plan and is “not sufficient if you start to go beyond that” – a point that is conceded in the water scarcity document. "

https://www.building.co.uk/focus/a-vision-for-150000-homes-b...

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#329
post #328
post #318

Earlier quoted context omitted.

I don't know how NY is in comparison, but housing in Cambridge is almost as expensive as in London. A detached 3br starts at £700k. NIMBYs keep killing any expansion of supply.

Some say NIMBY's other say water supply. "Water supply issues are already holding back housing development around the city. In its previous draft water resources management plan, Cambridge Water failed to demonstrate that there was enough to supply all of the new properties in the emerging local plan without risk of deterioration. The Environment Agency recently confirmed that it had formally objected to five large h…

The NIMBYs are the ones trying to stop the reservoir from being built. They created the problem they're complaining about. A big hole in the ground with water in it is not a complicated piece of infrastructure, but the planning system is so dysfunctional and veto-friendly that the construction timeline has been pushed out into the 2030s, in the best case. Previous generations got them done in two years flat. It is an artificial problem.

Same thing with transport. "We can't build new houses because it would increase car traffic", meanwhile putting up every barrier they can think of to stop East-West Rail.

Re: Effects of Gen AI on High Skilled Work: Experiments with Software Developers

#330

Reminds me of a situation I've been in a few times already: Dev: Hey einpoklum, how do I do XYZ? Me: Hmm, I think I remember that... you could try AB and then C. Dev: Ok, but isn't there a better/easier way? Let me ask ChatGPT. ... Dev: Hey einpoklum, ChatGPT said I should do AB and then C. Me: Let me have a look at that for a second. Me: -Right, so it's just what I read on StackOverflow about this, a couple of years…

> Sometimes it's even the answer that _I_ wrote on StackOverflow and then I feel cheated. What's next? Royalties from companies that used the answer in their solutions? Do you also want copyright of this comment?

> What's next?

Perhaps developing a little sense of humor? :-\

Post reply on HN