Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI coding tools can reduce productivity

secondthoughts.ai

Post reply on HN
Print view View original

151–160 of 284 posts

Re: AI coding tools can reduce productivity

#151

Post by tdiff »

tomcam wrote: post #16

What bothers me more than any of this particular discussion is that we seem to be incapable of determining programmer productivity in a meaningful way since my debut as a programmer 40 years ago.

But nevertheless, productivity objectively exists. Some people/teams are more productive as others.

I suppose it would be simpler to compare productivity for people working on standard, "normalized" tasks, but often every other task a programmer is assigned is something different to the previous one, and different developers get different tasks.

It's difficult to measure productivity based on real-world work, but we can create an artificial experiment: give N programmers the same M "normal", everyday tasks and observe whether those using AI tools complete them more quickly.

This is somewhat similar to athletic competitions — artificial in nature, yet widely accepted as a way to compare runners’ performance.

Re: AI coding tools can reduce productivity

#152

Post by computerthings »

jaredklewis wrote: post #55
tomcam wrote: post #16

What bothers me more than any of this particular discussion is that we seem to be incapable of determining programmer productivity in a meaningful way since my debut as a programmer 40 years ago.

I’m confused as to why anyone would think this would be possible to determine. Like can we determine the productivity of doctors, lawyers, journalists, or pastry chefs? What job out there is so simple that we can meaningfully measure all the positive and negative effects of the worker as well as account for different conditions between workers. I could probably get behind the idea that you could measure productivity…

[dead]

Re: AI coding tools can reduce productivity

#153

Post by Fraterkes »

I think the dichotomy you see with how positive people are about ai has almost entirely to do with the kind of questions they ask.

That seems obvious, but a consequence of that is that people who are sceptical of ai (like me) only use it when they've exhausted other resources (like google). You ask very specific questions where not a lot of documentation is available and inevetably even o3 ends up being pretty useless.

Conversely there's people who love ai and use it for everything, and since the majority of the stuff they ask about is fairly simple and well documented (eg "Write me some typescript"), they rarely have a negative experience.

Re: AI coding tools can reduce productivity

#154

Post by latexr »

AllegedAlec wrote: post #142
jaredklewis wrote: post #55

Earlier quoted context omitted.

I’m confused as to why anyone would think this would be possible to determine. Like can we determine the productivity of doctors, lawyers, journalists, or pastry chefs? What job out there is so simple that we can meaningfully measure all the positive and negative effects of the worker as well as account for different conditions between workers. I could probably get behind the idea that you could measure productivity…

> Like can we determine the productivity of doctors, lawyers, journalists, or pastry chefs? Yes, yes we can. Programmers really need to stop this cope about us being such special snowflakes that we can't be assessed and that our maangers just need to take that we're worth keeping around on good faith.

> Yes, yes we can.

Could you make an effort to explain how, or at the very least link to some reasoning? Otherwise your comment is basically the equivalent of “nuh-uh”, which doesn’t meaningfully contribute to the discussion.

> Programmers really need to stop this cope about us being such special snowflakes

Which is not at all what is happening in your parent comment. On the contrary, they’re putting developers on even footing with other professions.

Re: AI coding tools can reduce productivity

#155

Post by sheepfacts »

Perhaps is difficult to measure personal productivity in programming, but we can measure that we will run more slowly with 10 kg. in our backpack. I propose this procedure: The SWE selects 10 tasks and guesses some measure of their complexity (time to finish them) and then he randomly select 5 to be done with AI and the rest without. He performs them and finally calculates a deviation D. The deviation D = D_0 - D_1 where D_i = sum (real_time/guessed_time - 1), where D_0 is using AI and D_1 is without AI, the sign and magnitude of D measure respectively if the use of AI is beneficial or detrimental and the impact of using AI. Also, clipping individuals addends to be in the interval [-0.5,0.5] should avoid one bad guess dominating the estimation. Sorry if this is a trivial ideal but it is feasible and intuitively should provide useful information if the tasks are taken among the ones in which each initial guessing has small deviation. A filter should be applied to tasks in which scaffolding by AI surpass a certain relative threshold in case we are interested in generalizing our results to tasks in which scaffolding is not dominating time.

It could happen that the impact of using AI depends of the task at hand, the capability of the SWE to pair programming with it, and of the LLM used, to such an extend that those factors were bigger that the average effect on a bag of tasks, in this case the large deviation from the mean makes any one parameter estimation void of useful information.

Re: AI coding tools can reduce productivity

#156

Post by jaredklewis »

AllegedAlec wrote: post #142
jaredklewis wrote: post #55

Earlier quoted context omitted.

I’m confused as to why anyone would think this would be possible to determine. Like can we determine the productivity of doctors, lawyers, journalists, or pastry chefs? What job out there is so simple that we can meaningfully measure all the positive and negative effects of the worker as well as account for different conditions between workers. I could probably get behind the idea that you could measure productivity…

> Like can we determine the productivity of doctors, lawyers, journalists, or pastry chefs? Yes, yes we can. Programmers really need to stop this cope about us being such special snowflakes that we can't be assessed and that our maangers just need to take that we're worth keeping around on good faith.

News to me. How do you determine the productivity of a doctor? Patients seen? Patients cured? (for real, where did you get that data?) Number of medicines prescribed? Procedures performed? Does a triple bypass surgery count the same a pap smear? Hours worked? Amount of help they provided to colleagues? Easy to come up with another 100 other metrics that might be worth looking out. How are they all weighted?

Like I get that in SWE (like all other fields), managers have to make judgement calls and try to evaluate which reports contribute the most, but the GP post seemed surprised that this wasn't a solved problem by now, which just seems incomprehensible to me.

Re: AI coding tools can reduce productivity

#158

Post by tdiff »

thinkingemote wrote: post #99
calrain wrote: post #57

I've been using Claude Code heavily for about 3 months now, and I'm pretty sure I'm between 10 and 20 times more productive while using it. How I measure performance is how many features I can implement in a given period of time. It's nice that people have done studies and have opinions, but for me, it's 10x to 20x better.

I agree I feel more productive. AI tools do actually make it easier and makes my brain use less energy. You would think that would be more productive but maybe it just feels that way. Stage magicians say that the magic is done in the audiences memory after the trick is done. It's the effect of the activity. AI coding tools makes developers happier and able to spend more brain power on actually difficult things. But o…

> on actually difficult things

Can't help but note that in 99% cases this "difficult things" trope makes little sense. In most jobs, the freed time is either spent on other stupid tasks or is lost due to org inefficiencies, or is just procrastinated.

Re: AI coding tools can reduce productivity

#159

Post by latexr »

calrain wrote: post #57

I've been using Claude Code heavily for about 3 months now, and I'm pretty sure I'm between 10 and 20 times more productive while using it. How I measure performance is how many features I can implement in a given period of time. It's nice that people have done studies and have opinions, but for me, it's 10x to 20x better.

> I'm pretty sure

So were the people taking the study. Which is why we do these, to understand where our understanding of ourselves is lacking.

Maybe you are special and do get extra gains. Or maybe you are as wrong about yourself as everyone else and are overestimating the gains you think you have.

Re: AI coding tools can reduce productivity

#160

Post by JimDabell »

tomcam wrote: post #16

What bothers me more than any of this particular discussion is that we seem to be incapable of determining programmer productivity in a meaningful way since my debut as a programmer 40 years ago.

We can determine productivity for the purpose of studies like this. Give a bunch of developers the exact same task and measure how quickly they can produce a defect-free solution. Unfortunately, this study didn’t do that – the developers chose their own tasks.
Post reply on HN