Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

91–100 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#91

Post by taco_emoji »

ninetyninenine wrote: post #55
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

It’s not massively underplaying it imo. AI hype is real. This is revolutionary technology that humanity has never seen before. But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hyp…

Oh, buddy, LLM hallucinations are not the only gap left for AGI

Re: Being “Confidently Wrong” is holding AI back

#92

Post by chpatrick »

blueflow wrote: post #82
chpatrick wrote: post #22

Earlier quoted context omitted.

Nothing to do with dice though.

The whole "stochastic means to find factual correctness" thing is an error of method, arguing about weights here is nonsense.

It isn't though, the most factually correct human expert is also stochastic. The only question is how the dice are weighted.

Re: Being “Confidently Wrong” is holding AI back

#93

Post by dns_snek »

lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

> but in reality it simply plays into users' biases & makes it more likely that the user will accept & approve of incorrect responses from the AI.

Yes! I often find myself overthinking my phrasing to the nth degree because I've learned that even a sprinkle of bias can often make the LLM run in that direction even if it's not the correct answer.

It often feels a bit like interacting with a deeply unstable and insecure people pleasing person. I can't say anything that could possibly be interpreted as a disagreement because they'll immediately flip the script, I can't mention that I like pizza before asking them what their favorite food is because they'll just mirror me.

Re: Being “Confidently Wrong” is holding AI back

#94

Post by jqpabc123 »

rar00 wrote: post #57

I know people are pushing back, taking "only" literally, but from a reasonable perspective what causes LLMs (technically their outputs) to give that impression is indeed the crux of what holds progress back: how/what LLMs learn from data. In my personal opinion, there's something fundamentally flawed the whole field has yet to properly pinpointing and fix.

there's something fundamentally flawed the whole field has yet to properly pinpointing and fix.

Isn't it obvious?

It's all built around probability and statistics.

This is not how you reach definitive answers. Maybe the results make sense and maybe they're just nice sounding BS. You guess which one is the case.

The real catch --- if you know enough to spot the BS, you probably didn't need to ask the question in the first place.

Re: Being “Confidently Wrong” is holding AI back

#95

Post by blueflow »

chpatrick wrote: post #92
blueflow wrote: post #82

Earlier quoted context omitted.

The whole "stochastic means to find factual correctness" thing is an error of method, arguing about weights here is nonsense.

It isn't though, the most factually correct human expert is also stochastic. The only question is how the dice are weighted.

"human expert" as reference for "factually correct", oh just gently caress yourself. Appeal to authority (expert = social status) is as much bullshit as appeal to popularity.

Re: Being “Confidently Wrong” is holding AI back

#96

Post by chpatrick »

contagiousflow wrote: post #84
chpatrick wrote: post #26

Earlier quoted context omitted.

Sure, it's only as good as the training data. But human experts also output tokens with some statistical distribution. That doesn't mean anything.

If you think talking to an LLM is the same experience as talking to a human you should probably talk to more humans

That's not what I said. What I said is that the claim "LLMs aren't intelligent because they stochastically produce characters" doesn't hold because humans do that too even if they're intelligent and authorative.

Re: Being “Confidently Wrong” is holding AI back

#97

Post by jansper39 »

therobots927 wrote: post #45
traceroute66 wrote: post #32

Earlier quoted context omitted.

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

Honestly when I speak about these sorts of issues I get the feeling that other people view me as some kind of luddite, especially people above me who presumably want to replace as many people with AI as possible. I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

Re: Being “Confidently Wrong” is holding AI back

#98

Post by fuzzzerd »

gavinray wrote: post #69
therobots927 wrote: post #45

Earlier quoted context omitted.

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

> This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions.

While I agree, and also use your work around, I think it stands to reason this shouldn't be a problem. The context had the original problem statement along with several examples of what not to do and yet it keeps repeating those very things instead of coming up with a different solution. No human would keep trying one of the solutions included in the context that are marked as not valid.

Re: Being “Confidently Wrong” is holding AI back

#99

Post by z3c0 »

roxolotl wrote: post #11

The big thing here is that they can’t even be confident. There is no there there. They are a, admittedly very useful, statistical model. Ascribing confidence to it is an anthropomorphizing mistake which is easy to make since we’re wired to trust text that feels human. They are at their most useful when it is cheaper to verify their output than it is to generate it yourself. That’s why code is rather ok; you can run i…

Agreed. All these attempts to benchmark LLM performance based on the interpreted validity of the outputs are completely misguided. It may be the semantics of "context" causing people to anthropomorphize the models (besides the lifelike outputs.) Establishing context for humans is the process of holding external stimuli against an internal model of reality. Context for an LLM is literally just "the last n tokens". In that case, the performance would be how valid the most probablistic token was with the prior n tokens being present, which really has nothing to do with the perceived correctness of the output.

Re: Being “Confidently Wrong” is holding AI back

#100

Post by chpatrick »

JamesSwift wrote: post #80
Zigurd wrote: post #9

Earlier quoted context omitted.

The weights, so to speak, come from the knowledge base. That means you can't get away from the quality of the knowledge base. That isn't uniform across all domains of knowledge. Then the problem becomes how do you make the training material uniformly high-quality in every knowledge domain? At best it becomes the meta problem of determining the quality of knowledge in some way that makes an LLM able to calibrate confi…

I think youre missing the point. The issue is not the amount of knowledge it possesses. The problem is that theres no way to go from "statistically generate the next word" to "what is your confidence level in the fact you just stated". Maybe, with an enormous amount of computation we could layer another AI on top to evaluate or add confidence intervals, but I just dont see how we get there wihthout another quantum le…

Of course there is. If its training forces it to develop a theory of mind then it will weight the dice so that it's more likely to output "I don't know". Most likely the culprit is that it's hard to make training data for things that it doesn't know.
Post reply on HN