Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

121–130 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#121

Post by kilpikaarna »

ninetyninenine wrote: post #72
sfn42 wrote: post #36

Earlier quoted context omitted.

Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.

No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the ini…

> It is clear to everyone who builds LLMs that the AI is intelligent.

So presumably we have a solid, generally-agreed-upon definition on intelligence now?

> autocompleting things with humanity changing intelligent content.

What does this even mean?

Re: Being “Confidently Wrong” is holding AI back

#122

Post by gavinray »

dinfinity wrote: post #116
gavinray wrote: post #69

Earlier quoted context omitted.

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

I concur. Something to keep in mind is that it is often more robust to pull an LLM towards the right place than to push it away from the wrong place (or more specifically, the active parts of its latent space). Sidenote: also kind of true for humans. That means that positively worded instructions ("do x") work better than negative ones ("don't do y"). The more concepts that you don't want it to use / consider show up…

  >  positively worded instructions ("do x") work better than negative ones ("don't do y")
I've noticed this.

I saw someone on Twitter put it eloquently: something about how, just like little kids, the moment you say "DON'T DO XYZ" all they can think about is "XYZ..."

Re: Being “Confidently Wrong” is holding AI back

#123

Post by staticman2 »

chpatrick wrote: post #112
krapp wrote: post #109

Earlier quoted context omitted.

That doesn't really prove anything. I could create a Markov chain with a random seed that doesn't always answer the same question the same way, but that doesn't prove the human brain works like a Markov chain with a random seed. One thing humans tend not to do is confabulate entirely to the degree that LLMs do. When humans do so, it's considered a mental illness. Simply saying the same thing in a different way is not…

I'm not making any claim about how the human brain works. The only thing I'm saying is that humans also produce somewhat randomized output for the same question, which is pretty uncontroversial I think. That doesn't mean they're unintelligent. Same for LLMs.

I really wish people into LLMs would limit themselves to terms from neuroscience or philosophy when descrbing humans.

You are in my mind rightfully getting pushback for writing "human experts also output tokens with some statistical distribution. "

Re: Being “Confidently Wrong” is holding AI back

#124

Post by KoolKat23 »

lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

Gemini 2.5 pro is quite good at being stubborn (well at least the initial release versions, haven't tested since).

Re: Being “Confidently Wrong” is holding AI back

#126

Post by stetrain »

nyeah wrote: post #77
stetrain wrote: post #38

Earlier quoted context omitted.

Yep, the LLM will happily continue this spiral indefinitely but I've learned that if providing a bit more context and one correction doesn't provide a good solution, continuing is generally a waste of time. They tend to very quickly lose useful context of the original problem and stated goals.

Yes, that is the point of the comment.

Yes, you’re absolutely right! Agreeing with the comment and adding my own experience was the point of my comment.

Is there anything else I can help you with?

Re: Being “Confidently Wrong” is holding AI back

#127

Post by corytheboyd »

Isn’t it obvious that the confidently wrong problem will never go away because all of this is effectively built on a statistical next token matcher? Yeah sure you can throw on hacks like RAG, more context window, but it’s still built on the same foundation.

It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obviously shows when you take the 2D thing and turn it.

It seems like the effectiveness plateau of these hacks will soon be (has been?) reached and the smoke and mirrors snake oil sales booths cluttering Main Street will start to go away. Still a useful piece of tech, just, not for every-fucking-thing.

Re: Being “Confidently Wrong” is holding AI back

#128

Post by ACCount37 »

tliltocatl wrote: post #78
rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

> LLM doesn't learn incrementally from previous encounters This. Lack of any way to incorporate previous experience seems like the main problem. Humans are often confidently wrong as well - and avoiding being confidently wrong is actually something one must learn rather than an innate capability. But humans wouldn't repeat same mistake indefinitely.

You can gather feedback from inference and funnel that back into model training. It's just very, very hard to do that without shooting yourself in the foot.

The feedback you get is incredibly entangled, and disentangling it to get at the signals that would be beneficial for training is nowhere near a solved task.

Even OpenAI has managed to fuck up there - by accidentally training 4o to be a fully bootlickmaxxed synthetic sycophant. Then they struggled to fix that for a while, and only made good progress at that with GPT-5.

Re: Being “Confidently Wrong” is holding AI back

#130

Post by indigoabstract »

sfn42 wrote: post #36
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.

I think what matters most is that we now know that it's possible, that a computer mimicking most of our abilities (but not all) which we have long considered intelligent is obviously possible in some indeterminate future.

It's not obvious how long until that point or what form it will finally take, but it should be obvious that it's going to happen at some point.

My speculation is that until AI starts having senses like sight, hearing, touch and the ability to learn from experience, it will always be just a tool/help/aider to someone doing a job, but could not possibly replace that person in that job as it lacks the essential feedback mechanisms for successfully doing that job in the first place.

Post reply on HN