Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLM Critics Are Right. I Use LLMs Anyway

theocharis.dev

Post reply on HN
Print view View original

191–200 of 330 posts

Re: The LLM Critics Are Right. I Use LLMs Anyway

#191

Post by westurner »

exe34 wrote: post #160
westurner wrote: post #116

Earlier quoted context omitted.

Like becoming a manager? When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce. Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax Managers and developers are responsible for setting a test coverage threshold for merge approval. If there is 100% branch coverage…

I think what this fails to cover is that managers rely on other people to do that understanding and keeping things in mind. When there's a problem, the manager isn't just saying to a random person "go fix this", most of the time they're relying on somebody who has a deeper understanding of the problem and is not likely to randomly break something else just to fix this one bug.

Fearless refactoring is possible given test coverage? True in my experience; both with higher-risk human-developed software and with agent-coded software.

I haven't yet made myself learn the new swarm of concurrent agents with different specializations/agent_instructions methods yet.

Are multiple worktrees worth the cognitive burden and merge overhead?

A merge maintainer is always in code review mode

Re: The LLM Critics Are Right. I Use LLMs Anyway

#192

Post by coldtea »

westurner wrote: post #184
coldtea wrote: post #171

Earlier quoted context omitted.

I have been seeing such trivial shit getting submitted and deleting them from the test suites, but it's disheartening to see that the devs didn't have the better taste to see those LLM "tests" are so trivial they border on noop before submitting them...

What new or existing metric would better assess test coverage than line coverage and branch coverage? I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests.…

>What new or existing metric would better assess test coverage than line coverage and branch coverage?

Substance coverage - testing the actual logic, edge cases, etc. Not mere lines.

Re: The LLM Critics Are Right. I Use LLMs Anyway

#193

Post by alanwreath »

Diogenesian wrote: post #74
qsort wrote: post #62

Earlier quoted context omitted.

> I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM I think there are two different claims here: - developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims. - LLMs give no productivity uplift at all, which is much…

Specifically my claim is "the relatively minor productivity uplift I would personally get out of agentic development is offset by the high cost, along with unresolved questions about long-term code maintainability, so I am not convinced that it is actually beneficial." IMO the bigger problem is that ~1.5x individual dev productivity uplift seems to translate into 1.05x uplift across the team. People have been waaaaay…

Could it also be that the phrase “using an LLM” is loaded in itself? Who is getting more velocity out of it:

- chat box interaction

- vendor harness user (the cursor / antigravity crowd)

- SDD Claude users

- bespoke harness creators who let it goooooooo

I mean we already know who’s using all the tokens and driving enough cost to make penny pinchers consider removing other “costs”

Re: The LLM Critics Are Right. I Use LLMs Anyway

#194

Post by bigstrat2003 »

pigpop wrote: post #150
cj wrote: post #40

Earlier quoted context omitted.

Now imagine if your calculator billed per button press. And imagine you can't own a calculator because owning one outright requires too much hardware (or whatever).

This is just a repackaged argument against SaaS and cloud infrastructure. Every business could be using subscription free spreadsheet software and storing all of their data on prem and employing an IT team to manage the servers and data but the majority choose to pay a third party to handle all of that. Hell, they could even go so far as to do it all manually with written records and filing cabinets. All of those wor…

> This is just a repackaged argument against SaaS and cloud infrastructure.

Which, ironically enough, is still a terrible idea for a lot of businesses. But it's trendy, so they go with it.

Re: The LLM Critics Are Right. I Use LLMs Anyway

#195

Post by bigstrat2003 »

threethirtytwo wrote: post #93
adamas wrote: post #86

Earlier quoted context omitted.

With that logic, you would let an intern force push his way to prod, that's not a smart move.

no force pushes. And probably not an intern. But code that never gets looked at and only tested? Yes. And as AI gets better this will be happening in all companies... more and more.

Anyone who lets the LLM put out unreviewed code is shockingly negligent and bad at his job. It certainly might be the case that many people are bad in such a way, but that doesn't make it acceptable.

Re: The LLM Critics Are Right. I Use LLMs Anyway

#196

Post by ermantrout »

qsort wrote: post #62
Diogenesian wrote: post #47

Earlier quoted context omitted.

I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM productivity gains, which still seems true for agentic AI: https://metr.org/blog/2026-05-11-ai-usage-survey/ It is particularly striking how similar the results are to LLMs before agents. Along with the total absence of long-term data, I think the benefit can be (weakly) denied. Maybe not in the employmemt…

> I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM I think there are two different claims here: - developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims. - LLMs give no productivity uplift at all, which is much…

[flagged]

Re: The LLM Critics Are Right. I Use LLMs Anyway

#197

Post by ermantrout »

someoneiam wrote: post #149
pydry wrote: post #55

Earlier quoted context omitted.

The productivity depends upon the requirements. If slop is fine (and sometimes it is), the benefits are undeniable. If the dev was the kind that would have produced slop anyway - again, undeniable boost. If the quality needs to be high I think it actually can slow you down, though.

Agreed, but also to expand, if the dev is mediocre and ai-assisted coding is a skill, the productivity gains are 10x more mediocre code. Since code is a liability, this is not desirable. Hence, mediocre devs being more productive is an underestimated problem of the age of ai-assisted coding. I see this every day. The result is a whole bunch of dysfunctional systems unnecessarily dislodging perfectly acceptable proces…

"Code is a liability" is the part that usually gets hand-waved, and there's a number on it now. Veracode's 2025 GenAI report ran 80 tasks across 100+ models: when a task could be solved securely or insecurely, the models picked the insecure path 45% of the time, at roughly 2.7x the vulnerability density of human-written code (Java was worst, at a 72% failure rate).

So the mediocre-dev case may be worse than "10x more mediocre code." It's more code that also skews insecure by default, and that cost shows up downstream in review and incidents, not at the PR. Throughput goes up, and so does the liability per line.

Re: The LLM Critics Are Right. I Use LLMs Anyway

#198

Post by justthehuman »

msdz wrote: post #2

> LLM’s amplify what you already have: opinions, structure, frameworks. So far, so agreeable, but… > If you have thoughts, they come out sharper and faster. I can’t help but wonder whether constant use of “agent” harnesses will lead to an atrophy of the software engineering (or really any field) muscles. Actual muscles need exercise to stay in shape (let alone grow), so does the brain. Can we really be sure that thou…

[dead]

Re: The LLM Critics Are Right. I Use LLMs Anyway

#199

Post by tpdly »

skydhash wrote: post #112
qsort wrote: post #62

Earlier quoted context omitted.

> I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM I think there are two different claims here: - developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims. - LLMs give no productivity uplift at all, which is much…

> LLMs give no productivity uplift at all, which is much harder to defend It’s not really hard to defend. Because when people says that productivity is uplifted, they are talking about amount of work, not the ROI. That’s why you keep hearing about LOC, amount of PR and prototypes, and the time taken is actually “time to PR” and not “time to production + time spent on bugs”.

There's another even more abstract liability on the balance sheet here for "deprecated LOC" or something, where the thing they built gets scrapped because no one wants to untangle the rat nest, or the "documentation" was so incoherent that it all had to be rewritten and reviewed.

Re: The LLM Critics Are Right. I Use LLMs Anyway

#200

Post by FunHearing3443 »

msdz wrote: post #2

> LLM’s amplify what you already have: opinions, structure, frameworks. So far, so agreeable, but… > If you have thoughts, they come out sharper and faster. I can’t help but wonder whether constant use of “agent” harnesses will lead to an atrophy of the software engineering (or really any field) muscles. Actual muscles need exercise to stay in shape (let alone grow), so does the brain. Can we really be sure that thou…

I'm frankly baffled by the number of commenters saying they've seen little productivity gains using LLMs/agents. My team got cut in half at the end of last year and we've probably increased our output by 25% primarily due to LLMs. Personally it's probably doubled my productivity. And we generally use a pretty structured spec based approach with consistent human code review, and six months into adoption we've had virtually no bugs attributable to an AI mistake (in fact production issues have gone down, knock on wood). My point being we aren't vibe coding.

I speculate the gains must be heavily context dependent (no pun intended). Different teams, tech stacks, product types, and company policies perhaps have an outsized effect on what gains LLMs give. We are allowed to be pretty independent and don't have much of a restriction on token usage.

Post reply on HN