Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

181–190 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#181

Post by PessimalDecimal »

hodgehog11 wrote: post #159
roxolotl wrote: post #11

The big thing here is that they can’t even be confident. There is no there there. They are a, admittedly very useful, statistical model. Ascribing confidence to it is an anthropomorphizing mistake which is easy to make since we’re wired to trust text that feels human. They are at their most useful when it is cheaper to verify their output than it is to generate it yourself. That’s why code is rather ok; you can run i…

But as a statistical model, it should be able to report some notion of statistical uncertainty, not necessarily in its next-token outputs, but just as a separate measure. Unfortunately, there really doesn't seem to be a lot of effort going into this.

Even then, wouldn't its uncertainty be about the probability of the output given the input? That's different from probability of being correct in some factual sense. At least for this class of models.

Re: Being “Confidently Wrong” is holding AI back

#182

Post by Tiktaalik »

criddell wrote: post #60
lvl155 wrote: post #24

Earlier quoted context omitted.

Incrementally learning model is pretty hard. That’s actually something I am working on right now and it’s completely different from developing/implementing LLMs.

I think that's what it's going to take. Eventually put the learning model in a robot body and send it out into the real world where there's no shortage of training data.

Yea it'll learn real quick what falling in a ravine is like

Re: Being “Confidently Wrong” is holding AI back

#183

Post by dingnuts »

_flux wrote: post #162
vidarh wrote: post #139

Earlier quoted context omitted.

This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is w…

Indeed, arguing with LLM is good if you like arguing. For results it's not the way to go. I think often it's not required to completely start over: just identify the part where it goes off the rails, and modify your prompt just before that point. But yeah, basically the same process.

I don't know why this could be the case but I have absolutely gotten better results out of the bot after insulting it.

Re: Being “Confidently Wrong” is holding AI back

#184

Post by xienze »

ModernMech wrote: post #164
gavinray wrote: post #69

Earlier quoted context omitted.

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

> I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. This is going to age like "full self driving cars in 5 years". Yeah it'll gain capabilities, maybe it does do 80% of the work, but it still can't really drive itself, so it ultimately won't replace you like people are predicting. The money train assures that AGI/FSD will always be 6-18 months away, despit…

> The money train assures that AGI/FSD will always be 6-18 months away

I vividly remember when some folks from Microsoft come to my school to give a talk at some Computer Science event and proclaimed that yep, we have working AGI, the only limiting factor is hardware, but that should be resolved in about ten years.

This was in 2001.

Some grifts in technology are eternal.

Re: Being “Confidently Wrong” is holding AI back

#185

Post by ninetyninenine »

eCa wrote: post #90
ninetyninenine wrote: post #72

Earlier quoted context omitted.

No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the ini…

> Like you realize humans hallucinate too right? A developer that hallucinates at work to the extent that LLMs does would probably have issues getting their PRs past code reviews a lot.

A person who has schizophrenia and hallucinates to a greater extent than LLMs are clearly defective and not intelligent or sentient.

Because of this we should euthanize all schizophrenics. Just stab them to death or put a bullet in their heads right? I mean they aren’t intelligent or sentient so you shouldn’t feel anything when you do this.

I’m baffled as to why people think of this in terms of PRs. Like the LLM is intelligent but everyone’s like oh it’s not following my command perfectly therefore it’s not intelligent.

Re: Being “Confidently Wrong” is holding AI back

#186

Post by raynr »

corytheboyd wrote: post #127

Isn’t it obvious that the confidently wrong problem will never go away because all of this is effectively built on a statistical next token matcher? Yeah sure you can throw on hacks like RAG, more context window, but it’s still built on the same foundation. It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obv…

As a layman, this too strikes me as the problem underlying the "confidently wrong" problem.

The author proposes ways for an AI to signal when it is wrong and to learn from its mistakes. But that mechanism feeds back to the core next token matcher. Isn't this just replicating the problem with extra steps?

I feel like this is a framing problem. It's not that an LLM is mostly correct and just sometimes confabulates or is "confidently wrong". It's that an LLM is confabulating all the time, and all the techniques thrown at it do is increase the measured incidence of LLM confabulations matching expected benchmark answers.

Re: Being “Confidently Wrong” is holding AI back

#187

Post by more_corn »

gavinray wrote: post #69
therobots927 wrote: post #45

Earlier quoted context omitted.

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

Herald AI has a pretty robust mechanism for context cleanup. I think I saw a blogpost from them about it.

Re: Being “Confidently Wrong” is holding AI back

#188

Post by brookst »

traceroute66 wrote: post #32
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

You're conflating "correct themselves" with "are guaranteed to give the correct answer", which are two really different things. And in fact you're just echoing GP's point: their corrections can be wrong.

You case is no different from:

- AI: "The capital of France is Paris"

- User: "This is wrong, it changed to Montreal in 2005"

- AI: "You're absolutely right! The capital of France is Montreal"

Re: Being “Confidently Wrong” is holding AI back

#189

Post by ninetyninenine »

bigstrat2003 wrote: post #180
eCa wrote: post #90

Earlier quoted context omitted.

> Like you realize humans hallucinate too right? A developer that hallucinates at work to the extent that LLMs does would probably have issues getting their PRs past code reviews a lot.

They would have issues even remaining employed. AI defenders are very quick to point out "humans mistakes too", but that is a false equivalence because humans learn . If a junior makes a really stupid mistake, when I show him the correct way he won't make that mistake again. An AI will, because (as people correctly point out) it has no actual intelligence .

There’s examples of humans who can’t learn. Have you seen the movie memento.

There are cases where humans lose all ability to form long term memories and outside of a timed context window they remember nothing. That context window is minutes at best.

According to your logic these people have no actual intelligence or sentience. Therefore they should be euthanized. You personally can grab a gun and execute each of these people one by one with a bullet straight to the head because clearly these people have no actual intelligence or sentience. That’s the implication of your logic.

https://en.m.wikipedia.org/wiki/Anterograde_amnesia

It’s called anterograde amnesia. Do you see how your logic can justify gassing all these people holocaust style?

When I point out the flaw in your logic do you use the new facts to form a new conclusion? Or do you rearrange the facts to maintain support for your existing conclusion?

If you did the later I hate to tell you this, it wasn’t very intelligent. It was biased. But given that you’re human, that’s what you most likely did and it’s normal. But pause for a second and try to do the former of using the new facts to form a different more nuanced conclusion.

Re: Being “Confidently Wrong” is holding AI back

#190

Post by kovacs »

corytheboyd wrote: post #127

Isn’t it obvious that the confidently wrong problem will never go away because all of this is effectively built on a statistical next token matcher? Yeah sure you can throw on hacks like RAG, more context window, but it’s still built on the same foundation. It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obv…

This is the best analogy I've read to explain what's going on and takes me back to the days of Doom and how it was so transformative at the time. Perhaps in time the current generation will be viewed as the Doom engine as we await the holy grail of full 3D in Quake.
Post reply on HN