Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI coding tools can reduce productivity

secondthoughts.ai

Post reply on HN
Print view View original

181–190 of 284 posts

Re: AI coding tools can reduce productivity

#181

Post by Fraterkes »

rsanheim wrote: post #171
Fraterkes wrote: post #153

I think the dichotomy you see with how positive people are about ai has almost entirely to do with the kind of questions they ask. That seems obvious, but a consequence of that is that people who are sceptical of ai (like me) only use it when they've exhausted other resources (like google). You ask very specific questions where not a lot of documentation is available and inevetably even o3 ends up being pretty useles…

I don't think that dichotomy is true at all, at least not with experienced software people. Many folks I know are skeptical of the hype, or maybe full on anti/distrustful, due to reasons I think are valid. But many of those same people have tried llm tools, maybe chatgpt or copilot or cursor, and recognize the value even w/ huge misgivings. Some of have gone further with tools like claude code and seen the real poten…

I obviously don't mean that people literally write "write me some typescript", because nobody wants code that does something arbitrary. I'm also not saying that every reaction to ai falls between love and skeptical: I wrote a 3 sentence comment on a complex topic to sketch out an idea.

The tone of your comment suggests that my comment upset you, which wasn't my intent. But you have to try to be a little generous when you read other peoples stuff, or these discussion will get very tedious quickly.

Re: AI coding tools can reduce productivity

#182

Post by rcxdude »

latexr wrote: post #154
AllegedAlec wrote: post #142

Earlier quoted context omitted.

> Like can we determine the productivity of doctors, lawyers, journalists, or pastry chefs? Yes, yes we can. Programmers really need to stop this cope about us being such special snowflakes that we can't be assessed and that our maangers just need to take that we're worth keeping around on good faith.

> Yes, yes we can. Could you make an effort to explain how, or at the very least link to some reasoning? Otherwise your comment is basically the equivalent of “nuh-uh”, which doesn’t meaningfully contribute to the discussion. > Programmers really need to stop this cope about us being such special snowflakes Which is not at all what is happening in your parent comment. On the contrary, they’re putting developers on ev…

>Could you make an effort to explain how, or at the very least link to some reasoning? Otherwise your comment is basically the equivalent of “nuh-uh”, which doesn’t meaningfully contribute to the discussion.

You can look at the kind of work they're doing, how effective their solutions are, and how long it takes them to do it. That's the basics of it across a wide range of professions. Now, there's no one-size-fits-all metric or formula you can just calculate based on objective facts for most of this, because the work is more varied than e.g. factory work, but it's also not impossible to make the comparison, if you actually understand the work reasonably and you use judgement.

In the case of this study, because the assignment of the comparison they were doing was random, then just measuring time to completion across a range of tasks is a perfectly reasonable metric, because there's nothing to really bias the outcome, just a lot of factors that add noise instead. But it is worth noting that the result is a very broad average, and there is likely a very complicated distribution of details underneath, which is much harder to measure.

Re: AI coding tools can reduce productivity

#183

Post by specproc »

Tainnor wrote: post #10
freetime2 wrote: post #5

Here is the the methodology of the study: > To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors t…

The sample size isn't 16 developers, it's 246 issues.

Whilst my recent experience possibly agrees with the findings, I came here to moan about the methods. Whether it's 16 or 246, that's still a miserably small sample size.

Re: AI coding tools can reduce productivity

#184

Post by numpad0 »

jaredklewis wrote: post #156
AllegedAlec wrote: post #142

Earlier quoted context omitted.

> Like can we determine the productivity of doctors, lawyers, journalists, or pastry chefs? Yes, yes we can. Programmers really need to stop this cope about us being such special snowflakes that we can't be assessed and that our maangers just need to take that we're worth keeping around on good faith.

News to me. How do you determine the productivity of a doctor? Patients seen? Patients cured? (for real, where did you get that data?) Number of medicines prescribed? Procedures performed? Does a triple bypass surgery count the same a pap smear? Hours worked? Amount of help they provided to colleagues? Easy to come up with another 100 other metrics that might be worth looking out. How are they all weighted? Like I ge…

life expectancy, standardized qol metrics. or patients seen, revenue per patients, hours worked etc can be metrics if those _were_ what you wanted. the point is, the answer is yes, they have measures of better/worse docs in their field.

Re: AI coding tools can reduce productivity

#185

Post by alternatex »

greatwhitenorth wrote: post #101
calrain wrote: post #57

I've been using Claude Code heavily for about 3 months now, and I'm pretty sure I'm between 10 and 20 times more productive while using it. How I measure performance is how many features I can implement in a given period of time. It's nice that people have done studies and have opinions, but for me, it's 10x to 20x better.

I'm between 73 and 86 times more productive using claude code. You're not using it well.

Those are rookie numbers, you gotta pump those numbers up.

Re: AI coding tools can reduce productivity

#186

Post by alternatex »

philbo wrote: post #175
raggi wrote: post #24

They averaged producing 47% more code on the AI tasks, but took only 20% more time. The report here biases over these considerations, but I’m left wondering: was the extra code superfluous or did this produce better structure / managed debt better? If that extra 47% of code translates to lower debt and more consistent throughput over the long term, I might take it, given how crushed projects get from debt. Anyway, it…

I have an extremist take on this: All source code is technical debt. If you increase the amount of code, you increase the amount of debt. It's impossible to reduce debt with more code. The only way to reduce debt is by reducing code. (and note that I'm not measuring code in bytes here; switching to single-character variable names would not reduce debt. I'm measuring it in statements, expressions, instructions; reduci…

I'll try a counterargument. If more code is more technical debt then writing more succinct code is less technical debt. But succinct code is often harder to grok and maintain than code written for the average Joe dev. So less code can sometimes mean less maintainability and thus more technical debt.

I think you instead meant to say more business logic implemented in code is more technical debt, not necessarily just more code.

Re: AI coding tools can reduce productivity

#187

Post by bheadmaster »

AllegedAlec wrote: post #164
bheadmaster wrote: post #147

Earlier quoted context omitted.

> Yes, yes we can. Of course we can . But can we do it in a meaningful way, such that the metric itself doesn't become a subject to optimization? "When a measure becomes a target, it ceases to be a good measure"

> the metric itself doesn't become a subject to optimization By making the metrics part of a sustaintable company-wide goal. If there's a company-wide goal to increase X kind of revenue by Y% making actionable targets on how a team can contribute (not lazy shit like "our changes should contribute Z% of that Y%"), and within that create for a person another smaller metric based on that.

It's all so great in theory, where you get to imagine an ideal world in which all our incentives align and we're all rowing on a big boat towards success.

In real world, most things don't work out that way. What metrics do you use to measure surgeons' success? If you use fatality rate, then as a result surgeons will refuse to do more risky surgeries which will put their ratings at risk, which makes the healthcare worse, instead of better.

Re: AI coding tools can reduce productivity

#188

Post by yakattak »

jack_pp wrote: post #81
freetime2 wrote: post #5

Here is the the methodology of the study: > To directly measure the real-world impact of AI tools on software development, we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors t…

I think the productivity gains most people rave about are stuff like, I wanted to do X which isn't hard if you are experienced with library Y and library Y is pretty popular and the LLM did it perfectly first try! I think that's where you get 10-20x. When you're working on niche stuff it's either not gonna work or work poorly. For example right now I need to figure out why an ffmpeg filter doesn't do X thing smoothly…

I really think they excel at greenfield work, and are “fine” at writing code for existing systems. When you are unfamiliar with a library or a pattern it’s a huge time saver.

Re: AI coding tools can reduce productivity

#189

Post by bheadmaster »

AllegedAlec wrote: post #165
jaredklewis wrote: post #156

Earlier quoted context omitted.

News to me. How do you determine the productivity of a doctor? Patients seen? Patients cured? (for real, where did you get that data?) Number of medicines prescribed? Procedures performed? Does a triple bypass surgery count the same a pap smear? Hours worked? Amount of help they provided to colleagues? Easy to come up with another 100 other metrics that might be worth looking out. How are they all weighted? Like I ge…

> How do you determine the productivity of a doctor? At the end of the road. Patient outcome and contentedness compared to others with similar indications. Patients seen and all that is that sort of short-term BS that you see everywhere that's giving metrics a bad name. It'd be like determining a mechanic's productivity by how many times he twisted a wrench.

> Patient outcome and contentedness compared to others with similar indications

Which would incentivise doctors to refuse to treat patients who are more ill, lest they risk their ratings go down.

Re: AI coding tools can reduce productivity

#190

Post by [deleted] »

aleph_minus_one wrote: post #169
Fraterkes wrote: post #153

I think the dichotomy you see with how positive people are about ai has almost entirely to do with the kind of questions they ask. That seems obvious, but a consequence of that is that people who are sceptical of ai (like me) only use it when they've exhausted other resources (like google). You ask very specific questions where not a lot of documentation is available and inevetably even o3 ends up being pretty useles…

I think there are also other aspects: - Some people simply ask a lot more questions than others (this ignores whether they like or dislike AI), i.e. some people simply prefer to find things out more by themselves, and thus also use other resources like Google or Stack Overflow as a last resort. So their questions to an AI will likely be more complicated, because they already found out the easy parts by themselves. -…

[deleted]
Post reply on HN