Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

221–230 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#221

Post by vidarh »

xienze wrote: post #178
vidarh wrote: post #139

Earlier quoted context omitted.

This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is w…

> I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer Maybe because people expect AI systems that are touted as all-knowing, all-powerful, coming-for-your-job to be smart enough to remember what was said two turns ago?

That's fine once or twice. At that point people should learn that this isn't how they work, and figure out how to use them better.

It's not a tools fault if people insist on continuing to use them in counter-productive ways.

Re: Being “Confidently Wrong” is holding AI back

#222

Post by Wowfunhappy »

dinfinity wrote: post #116
gavinray wrote: post #69

Earlier quoted context omitted.

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

I concur. Something to keep in mind is that it is often more robust to pull an LLM towards the right place than to push it away from the wrong place (or more specifically, the active parts of its latent space). Sidenote: also kind of true for humans. That means that positively worded instructions ("do x") work better than negative ones ("don't do y"). The more concepts that you don't want it to use / consider show up…

> That means that positively worded instructions ("do x") work better than negative ones ("don't do y").

In teacher school, we're told to always give kids affirmative instructions, ie "walk" instead of "don't run". The idea is that it takes more energy for a child to figure out what to do.

Re: Being “Confidently Wrong” is holding AI back

#223

Post by esafak »

z3c0 wrote: post #175
hodgehog11 wrote: post #159

Earlier quoted context omitted.

But as a statistical model, it should be able to report some notion of statistical uncertainty, not necessarily in its next-token outputs, but just as a separate measure. Unfortunately, there really doesn't seem to be a lot of effort going into this.

The statistical certainty is indeed present in the model. Each token comes with a probablility; if your softmax results approach a uniform distribution (i.e. all selected tokens at the given temp have near equal probabilities), then the next most likely token is very uncertain. Reporting the probabilities of the returned tokens can help the user understand how likely hallucinations are. However, that information is d…

That is not the same thing! You are talking about the point distribution of the next token. We are talking about the uncertainty associated with each of those candidate tokens; a distribution of distributions.

It's the difference between a categorical distribution and a Dirichlet. https://en.wikipedia.org/wiki/Dirichlet_distribution

Re: Being “Confidently Wrong” is holding AI back

#224

Post by jrm4 »

Genuine question from someone who thinks they understand the tech:

I don't get why I haven't seen a whole lot of (or any) of these models or tools "self reporting" on "confidence in their answer?"

This feels like it would be REALLY easy; these things predict likelihoods of tokens -- just, you know, give us that number?

Re: Being “Confidently Wrong” is holding AI back

#225

Post by ninetyninenine »

taco_emoji wrote: post #91
ninetyninenine wrote: post #55

Earlier quoted context omitted.

It’s not massively underplaying it imo. AI hype is real. This is revolutionary technology that humanity has never seen before. But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hyp…

Oh, buddy, LLM hallucinations are not the only gap left for AGI

It is. After that it's virtually indistinguishable from chatting with a human

Re: Being “Confidently Wrong” is holding AI back

#226

Post by z3c0 »

esafak wrote: post #223
z3c0 wrote: post #175

Earlier quoted context omitted.

The statistical certainty is indeed present in the model. Each token comes with a probablility; if your softmax results approach a uniform distribution (i.e. all selected tokens at the given temp have near equal probabilities), then the next most likely token is very uncertain. Reporting the probabilities of the returned tokens can help the user understand how likely hallucinations are. However, that information is d…

That is not the same thing! You are talking about the point distribution of the next token. We are talking about the uncertainty associated with each of those candidate tokens; a distribution of distributions. It's the difference between a categorical distribution and a Dirichlet. https://en.wikipedia.org/wiki/Dirichlet_distribution

I think we're talking about the same thing. I should be clear that I don't think the selected token probabilities being reported are enough, but if you're reporting each returned tokens probability (both selected and discarded) and aggregating the cumulative probabilities of the given context, it should be possible to see when you're trending centrally towards uncertainty.

Re: Being “Confidently Wrong” is holding AI back

#227

Post by asadotzler »

vidarh wrote: post #219
recursive wrote: post #197

Earlier quoted context omitted.

This is also a thing that's bad about LLMs. You're holding it wrong if you continue to argue. But LLMs are presented as if we can use the conventions of natural language to communicate with them. That's how they're sold. So if they fail to live up to those expectations, that's still a problem with LLMs.

It's a problem with LLM's and people are "holding it wrong". It makes zero difference that they've been sold as doing better if other people learn how to use them effectively and I choose to ignore how to get the best possible results out of them.

Except that it's impossible to "hold it right" -- even when following the guidance from its makers.

Re: Being “Confidently Wrong” is holding AI back

#228

Post by asadotzler »

vidarh wrote: post #221
xienze wrote: post #178

Earlier quoted context omitted.

> I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer Maybe because people expect AI systems that are touted as all-knowing, all-powerful, coming-for-your-job to be smart enough to remember what was said two turns ago?

That's fine once or twice. At that point people should learn that this isn't how they work, and figure out how to use them better. It's not a tools fault if people insist on continuing to use them in counter-productive ways.

It's not the tools fault when people RTFM (guidance from the tool maker) and use it as it's intended (again, by the tool maker, who presumably knows how it works and is in the best position to guide users).

"If you keep pressing the back button like the IE engineers told you to, of course you will fail to go back. To go back you want to press the forward button. Are you an idiot? Press the forward button to go back, at least until the next version release when you will need to press the reload button to go back. Trust me, eventually the back button will go back, but for now only fools press the back button to go back."

Re: Being “Confidently Wrong” is holding AI back

#229

Post by asadotzler »

vidarh wrote: post #141
therobots927 wrote: post #45

Earlier quoted context omitted.

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

Because it's easy to learn to stop engaging with those loops, treating them as a sign you provided too little context, and instead start a new conversation with an expanded prompt. It doesn't mean these loops aren't an issue, because they are, but once you stop engaging with them and cut them off, they're a nuisance rather than a showstopper.

They happen in subtle ways that aren't always easy and are rarely early in a project I want to just throw away.

"So what if you have to throw out a week's worth of work. That's how these things work. Accept it and you'll be happier. I have and I'm happy. Don't you see that it's OK to have your tool corrupt your work half way through. It's the future of work and you're being left behind by not letting your tools corrupt your work arbitrarily. Just start over like a real man."

Re: Being “Confidently Wrong” is holding AI back

#230

Post by xienze »

vidarh wrote: post #221
xienze wrote: post #178

Earlier quoted context omitted.

> I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer Maybe because people expect AI systems that are touted as all-knowing, all-powerful, coming-for-your-job to be smart enough to remember what was said two turns ago?

That's fine once or twice. At that point people should learn that this isn't how they work, and figure out how to use them better. It's not a tools fault if people insist on continuing to use them in counter-productive ways.

They’re non-deterministic, remember? So it’s not always the case that an LLM will get stuck in this sort of loop. Hence why people get frustrated when it happens and continue to think that perhaps it should be working on a more consistent basis.
Post reply on HN