[dead]
Being “Confidently Wrong” is holding AI back
51–60 of 274 posts
Re: Being “Confidently Wrong” is holding AI back
#52Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
> Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. Ha, that almost seems like an oxymoron. The previous encounters can be the new training data!
Queries are questions in a sense that they are not the original facts. I don’t think they are useful for training data.
Re: Being “Confidently Wrong” is holding AI back
#53I don't think humans are good at assessing the accuracy of their own opinions either and I'm not sure how AI is going to do it. Usually what corrects us is failure: some external stimulus that is indifferent or hostile to us.
As Mazer Rackham from Ender's Game said: "Only the enemy shows you where you are weak."
Re: Being “Confidently Wrong” is holding AI back
#54Earlier quoted context omitted.
> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…
Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most
The AI-fanbois will quickly tell you that you are misusing the context or your prompt is "wrong".
But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script).
And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use a specific version of a specific library ... and it goes and ignores me completely and picks some random library).
Re: Being “Confidently Wrong” is holding AI back
#55While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…
It’s not massively underplaying it imo. AI hype is real. This is revolutionary technology that humanity has never seen before.
But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hype. But inevitably like landing on the moon, humanity is suffering from hype fatigue.
Too much hype makes us numb to the reality of how insane the technology is.
Like when someone says the only thing stopping LLMs is hallucinations… that is literally the last gap. LLMs cover creativity, comprehension, analysis, knowledge and much more. Hallucinations is it. The final problem is targeted and boxed into something much more narrower then just build a human level AI from scratch.
Don’t get me wrong. Hallucinations are hard. But this being the last thing left is not an underplay. Yes it’s a massive issue but yes it is also a massive achievement to reduce all of agi to simply solving just an hallucination problem.
Re: Being “Confidently Wrong” is holding AI back
#56The link is a sales pitch for some tech that uses MCPs ... see the platform overview on the product top menu
Because MCPs solve the exact issue the whole post is about
Re: Being “Confidently Wrong” is holding AI back
#57I know people are pushing back, taking "only" literally, but from a reasonable perspective what causes LLMs (technically their outputs) to give that impression is indeed the crux of what holds progress back: how/what LLMs learn from data. In my personal opinion, there's something fundamentally flawed the whole field has yet to properly pinpointing and fix.
Re: Being “Confidently Wrong” is holding AI back
#58I don't think humans are good at assessing the accuracy of their own opinions either and I'm not sure how AI is going to do it. Usually what corrects us is failure: some external stimulus that is indifferent or hostile to us. As Mazer Rackham from Ender's Game said: "Only the enemy shows you where you are weak."
Maybe AI isn't artificial enough here...
Re: Being “Confidently Wrong” is holding AI back
#59Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
[dead]
Re: Being “Confidently Wrong” is holding AI back
#60Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
Incrementally learning model is pretty hard. That’s actually something I am working on right now and it’s completely different from developing/implementing LLMs.
I think that's what it's going to take. Eventually put the learning model in a robot body and send it out into the real world where there's no shortage of training data.