PG pointed this out a while back. He said that AIs were great at generating typical online comments. (NB I don't know which site's comments he might have been referring to.)
Being “Confidently Wrong” is holding AI back
71–80 of 274 posts
Re: Being “Confidently Wrong” is holding AI back
#72While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…
Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.
No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the initial fever pitched alarmist reaction people had to LLMs when it first came out.
Like you realize humans hallucinate too right? And that there are humans that have a disease that makes them hallucinate constantly.
Hallucinations don’t preclude humans from being “intelligent”. It also doesn’t preclude the LLM from being intelligent.
Re: Being “Confidently Wrong” is holding AI back
#73Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
Incrementally learning model is pretty hard. That’s actually something I am working on right now and it’s completely different from developing/implementing LLMs.
Cool, got any previous work to share?
Re: Being “Confidently Wrong” is holding AI back
#74Earlier quoted context omitted.
> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…
Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most
But still under pressure in the short-term, no? As companies lean into AI as a means of efficiency / competitive advantage / cost savings, jobs will be eliminated / reduced while companies find their direction. The potential gains are said to be too big to sit on the sidelines and wait to be a late-adopter.
Re: Being “Confidently Wrong” is holding AI back
#75Earlier quoted context omitted.
Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most
The AI-fanbois will quickly tell you that you are misusing the context or your prompt is "wrong". But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script). And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use…
I'd be curious if you could share some poor-performing prompts.
I would be willing to record myself using them across paid models with custom instructions and see if the output is still garbage.
Re: Being “Confidently Wrong” is holding AI back
#76I’m especially surprised by how little progress has been made. Today’s hallucinations, while less frequent, continue to have a major negative impact. And the problem has been noticed since the start.
> "I will admit, to my slight embarrassment … when we made ChatGPT, I didn't know if it was any good," said Sutskever.
> "When you asked it a factual question, it gave you a wrong answer. I thought it was going to be so unimpressive that people would say, 'Why are you doing this? This is so boring!'" he added.
https://www.businessinsider.com/chatgpt-was-inaccurate-borin...
Re: Being “Confidently Wrong” is holding AI back
#77Earlier quoted context omitted.
> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…
Yep, the LLM will happily continue this spiral indefinitely but I've learned that if providing a bit more context and one correction doesn't provide a good solution, continuing is generally a waste of time. They tend to very quickly lose useful context of the original problem and stated goals.
Yes, that is the point of the comment.
Re: Being “Confidently Wrong” is holding AI back
#78Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
> LLM doesn't learn incrementally from previous encounters
This. Lack of any way to incorporate previous experience seems like the main problem. Humans are often confidently wrong as well - and avoiding being confidently wrong is actually something one must learn rather than an innate capability. But humans wouldn't repeat same mistake indefinitely.
Re: Being “Confidently Wrong” is holding AI back
#79Anybody remember active learning? I'm old, and ML was much different back then, but this reminds me of grueling annotation work I had to do.
On a different note: is it just me or are some parts of this article oddly written? The sentence structure and phrasing read as confusing - which I find ironic, given the context.
Re: Being “Confidently Wrong” is holding AI back
#80Earlier quoted context omitted.
It is if the weights are sufficiently advanced.
The weights, so to speak, come from the knowledge base. That means you can't get away from the quality of the knowledge base. That isn't uniform across all domains of knowledge. Then the problem becomes how do you make the training material uniformly high-quality in every knowledge domain? At best it becomes the meta problem of determining the quality of knowledge in some way that makes an LLM able to calibrate confi…
I think youre missing the point. The issue is not the amount of knowledge it possesses. The problem is that theres no way to go from "statistically generate the next word" to "what is your confidence level in the fact you just stated". Maybe, with an enormous amount of computation we could layer another AI on top to evaluate or add confidence intervals, but I just dont see how we get there wihthout another quantum leap.