LLMs are a 400-year-long confidence trick

Are they now?

OpenAI's o3 was SOTA, and valued by its users for its high performance on hard tasks - while also being an absolute hallucination monster due to one of OpenAI's RLVR oopsies. You'd never know whether it's brilliant or completely full of shit at any given moment in time. People still used o3 because it was well worth it.

So clearly, hallucinations do not stop AI usage - or even necessarily undermine AI performance.

And if the bar you have to clear is "human performance", rather than something like "SQL database", then the bar isn't that high. See: the notorious unreliability of eyewitness testimonies.

Humans avoid hallucinations better than LLMs do - not because they're fundamentally superior, but because they get a lot of meta-knowledge "for free" as a part of their training process.

LLMs get very little meta-knowledge in pre-training, and little skill in using what they have. Doesn't mean you can't train them to be more reliable - there are pipelines for that already. It just makes it hard.

The marketing hype is economy defining at this point, so calling it overblown is an understatement.

Simplifying the hype into 2 threads, the first is that AI is an existential risk and the second is the promise of “reliable intelligence”.

The second is the bugbear, and the analogy I use is factories and assembly lines vs power tools.

LLMs are power tools. They are being hyped as factories of thoughts.

String the right tool calls, agents, and code together and you have an assembly line that manufactures research reports, gives advice, or whatever white collar work you need. No Holidays, HR, work hours, overhead etc.

I personally want everyone who can see why this second analogy does not work, to do their part in disabusing people of this notion.

LLMs are power tools, and impressive ones at that. In the right hands, they can do much. Power tools are wildly useful. But Power tools do not make automatically make someone a carpenter. They don’t ensure you’ve built a house to spec. Nor is a planar saw going to evolve into a robot.

The hype needs to be taken to task, preferably clinically, so that we know what we are working with, and can use them effectively.

The best way to describe AI agents (coding agents here) I heard on some presentation, I think it was from Andrej Karpathy.

It was something like this:

"We think we are building Ultron but really we are building the Iron Man suit. It will be a technology to amplify humans, not replace them"

Sure but increased output would mean code. I don't think generating a lot of code is itself developer productivity. Some people could be using it to stop themselves from creating bad code which is developer productivity. While I find it a bit unlikely people are using it in this way (in terms of the average) I would most certainly have made this argument if code quantity was up from LLMs so I can't claim to know a quantitative measure.

"My belief in this tech isn't based on marketing hype or someone telling me it's good - it's based on cold reality of what I'm shipping daily."

This may be true. The commenter may "believe in this tech" based on his experimentation with it

But the majority of sentences following this statement ironically appear to be "marketing hype" or "someone telling [us] it's good":

1. "The productivity gains I'm seeing right now are unprecedented."

2. "Even a year ago this wouldn't have been possible, it really feels like an inflection point."

3. "I'm seeing legitimate 10x gains because I'm not writing code anymore - I'm thinking about code and reading code."

4. "Using Claude Code Opus 4.5 right now and it's insane."

5. "It's like being a writer after Gutenberg invented the printing press rather than the monk copying books by hand before it."

The "framing" in this blog post is not focused on whether "this tech" actually saves anyone any time or money

It is focused on _hype_, namely how "this tech" is promoted. That promotion could be intentional or unintentional

N.B. I am not "agreeing" with the blog post author or "disagreeing" with the HN commenter, or vice versa. The point I'm making is that one is focused on whether "this tech" works for them and the other is focused on how "this tech" is being promoted. Those are two different things, as other replies have also noted. Additionally, the comment appears to be an example of the promotion (hype) that its author claims is not the basis for his "belief in this tech"

I think the use of the term "belief" is interesting

That term normally implies a lack of personal knowledge:

151 "Belief" gcide "The Collaborative International Dictionary of English v.0.48"

Belief \Be*lief"\, n. [OE. bileafe, bileve; cf. AS. gele['a]fa. See {Believe}.]

1. Assent to a proposition or affirmation, or the acceptance of a fact, opinion, or assertion as real or true, without immediate personal knowledge; reliance upon word or testimony; partial or full assurance without positive knowledge or absolute certainty; persuasion; conviction; confidence; as, belief of a witness; the belief of our senses. [1913 Webster]

Belief admits of all degrees, from the slightest suspicion to the fullest assurance. --Reid. [1913 Webster]

2. (Theol.) A persuasion of the truths of religion; faith. [1913 Webster]

No man can attain [to] belief by the bare contemplation of heaven and earth. --Hooker. [1913 Webster]

4. A tenet, or the body of tenets, held by the advocates of any class of views; doctrine; creed. [1913 Webster]

In the heat of persecution to which Christian belief was subject upon its first promulgation. --Hooker. [1913 Webster]

{Ultimate belief}, a first principle incapable of proof; an intuitive truth; an intuition. --Sir W. Hamilton. [1913 Webster]

Syn: Credence; trust; reliance; assurance; opinion. [1913 Webster]

151 "belief" wn "WordNet (r) 3.0 (2006)"

belief

n 1: any cognitive content held as true [ant: {disbelief}, {unbelief}]

2: a vague idea in which some confidence is placed; "his impression of her was favorable"; "what are your feelings about the crisis?"; "it strengthened my belief in his sincerity"; "I had a feeling that she was lying" [syn: {impression}, {feeling}, {belief}, {notion}, {opinion}]

151 "BELIEF" bouvier "Bouvier's Law Dictionary, Revised 6th Ed (1856)"

BELIEF. The conviction of the mind, arising from evidence received, or from information derived, not from actual perception by our senses, but from. the relation or information of others who have had the means of acquiring actual knowledge of the facts and in whose qualifications for acquiring that knowledge, and retaining it, and afterwards in communicating it, we can place confidence. " Without recurring to the books of metaphysicians' "says Chief Justice Tilghman, 4 Serg. & Rawle, 137, "let any man of plain common sense, examine the operations of, his own mind, he will assuredly find that on different subjects his belief is different. I have a firm belief that, the moon revolves round the earth. I may believe, too, that there are mountains and valleys in the moon; but this belief is not so strong, because the evidence is weaker." Vide 1 Stark. Ev. 41; 2 Pow. Mortg. 555; 1 Ves. 95; 12 Ves. 80; 1 P. A. Browne's R 258; 1 Stark. Ev. 127; Dyer, 53; 2 Hawk. c. 46, s. 167; 3 Wil. 1, s. 427; 2 Bl. R. 881; Leach, 270; 8 Watts, R. 406; 1 Greenl. Ev. Sec. 7-13, a.