Being “Confidently Wrong” is holding AI back
promptql.io
Being “Confidently Wrong” is holding AI back
1–10 of 274 posts
Re: Being “Confidently Wrong” is holding AI back
#2Rolling weighted dice repeatedly to generate words isn't factually accurate. More at 11.
Re: Being “Confidently Wrong” is holding AI back
#3Not just AI.
Re: Being “Confidently Wrong” is holding AI back
#4Rolling weighted dice repeatedly to generate words isn't factually accurate. More at 11.
It is if the weights are sufficiently advanced.
Re: Being “Confidently Wrong” is holding AI back
#5The interesting question here is if a statistical model like GPTs actually can encode this is a meaningful way. Nobody has quite found it yet, if so
Re: Being “Confidently Wrong” is holding AI back
#6Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5).
I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
Re: Being “Confidently Wrong” is holding AI back
#7Re: Being “Confidently Wrong” is holding AI back
#8While the thrust of this article is generally correct, I have two issues with it:
1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing.
2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate solution), but in reality it simply plays into users' biases & makes it more likely that the user will accept & approve of incorrect responses from the AI. Often, rather than "correcting" itself it merely "teaches" the AI how to be confidently wrong in an amenable & subtle manner which the individual user finds easy to accept (or more difficult to spot).
If anything, unless/until we can solve the (insurmountable) problem of AI being wrong, AI should at least be trained to be confidently & stubbornly wrong (or right). This would also likely lead to better consistency in testing.
Re: Being “Confidently Wrong” is holding AI back
#9Rolling weighted dice repeatedly to generate words isn't factually accurate. More at 11.
It is if the weights are sufficiently advanced.
The weights, so to speak, come from the knowledge base. That means you can't get away from the quality of the knowledge base. That isn't uniform across all domains of knowledge. Then the problem becomes how do you make the training material uniformly high-quality in every knowledge domain? At best it becomes the meta problem of determining the quality of knowledge in some way that makes an LLM able to calibrate confidence to a knowledge domain. But more likely we're stuck with the dubious quality that comes from human bias and wishful thinking in supposedly authoritative material.
Re: Being “Confidently Wrong” is holding AI back
#10Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
[deleted]