Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

21–30 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#21

Post by BitsAndObjects »

tango12 wrote: post #15
rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Author here. You’re right in that it’s obviously not the only problem. But without solving this seems like no matter how good the models get it’ll never be enough. Or, yes, the biggest research breakthrough we need is reliable calibrated confidence. And that’ll allow existing models as they are to become spectacularly more useful.

The biggest breakthrough that we need is something resembling actual intelligence in AI (human or machine, I’ll let you decide where we need it more ;) )

Re: Being “Confidently Wrong” is holding AI back

#22

Post by chpatrick »

blueflow wrote: post #7
chpatrick wrote: post #4

Earlier quoted context omitted.

It is if the weights are sufficiently advanced.

I find such statements frightening. Too many people can not tell the different between prevalence ("everybody does it") and factually correct.

Nothing to do with dice though.

Re: Being “Confidently Wrong” is holding AI back

#23

Post by moduspol »

rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Unclear limits on how much context can be reliably provided and effectively used without degrading the result.

Re: Being “Confidently Wrong” is holding AI back

#24

Post by lvl155 »

rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Incrementally learning model is pretty hard. That’s actually something I am working on right now and it’s completely different from developing/implementing LLMs.

Re: Being “Confidently Wrong” is holding AI back

#26

Post by chpatrick »

Zigurd wrote: post #9
chpatrick wrote: post #4

Earlier quoted context omitted.

It is if the weights are sufficiently advanced.

The weights, so to speak, come from the knowledge base. That means you can't get away from the quality of the knowledge base. That isn't uniform across all domains of knowledge. Then the problem becomes how do you make the training material uniformly high-quality in every knowledge domain? At best it becomes the meta problem of determining the quality of knowledge in some way that makes an LLM able to calibrate confi…

Sure, it's only as good as the training data. But human experts also output tokens with some statistical distribution. That doesn't mean anything.

Re: Being “Confidently Wrong” is holding AI back

#27

Post by lazide »

rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

The article is the peak of confidently wrong itself, for solid irony points.

Re: Being “Confidently Wrong” is holding AI back

#28

Post by ColinEberhardt »

I agree with the overall sentiment here, having written something similar recently:

“LLMs don’t know what they don’t know” https://blog.scottlogic.com/2025/03/06/llms-dont-know-what-t...

But I wouldn’t say it is the only problem with this technology! Rather, it is a subtle issue that most users don’t understand

Re: Being “Confidently Wrong” is holding AI back

#29

Post by projektfu »

roxolotl wrote: post #11

The big thing here is that they can’t even be confident. There is no there there. They are a, admittedly very useful, statistical model. Ascribing confidence to it is an anthropomorphizing mistake which is easy to make since we’re wired to trust text that feels human. They are at their most useful when it is cheaper to verify their output than it is to generate it yourself. That’s why code is rather ok; you can run i…

The article buries the lede by waiting until the very end to talk about solutions like having the LLM write DSL code. Presumably if you feed an LLM your orders table and a question about it, you'll get an answer that you can't trust. But if you ask it to write some SQL or similar thing based on your database to get the answer and run it, you can have more confidence.

Re: Being “Confidently Wrong” is holding AI back

#30

Post by paul7986 »

And being overHyped with the doom and gloom of it's affects on society.

chatGPT (5) is not there especially in replacing my field and skills: graphic, web design and web development. The first 2 there it spits out solid creations per your prompt request yet can not edit it's creations just creates new ones lol. So it's just another tool in my arsenal not a replacement to me.

Such Makes me wonder how it generates the logos and website designs ... is it all just hocus pocus.. the Wizard of OZ?

Post reply on HN