Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

211–220 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#211

Post by zparky »

gavinray wrote: post #69
therobots927 wrote: post #45

Earlier quoted context omitted.

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

Um, how much are you spending on running all these at once?

Re: Being “Confidently Wrong” is holding AI back

#213

Post by NoGravitas »

ninetyninenine wrote: post #55
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

It’s not massively underplaying it imo. AI hype is real. This is revolutionary technology that humanity has never seen before. But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hyp…

> Like when someone says the only thing stopping LLMs is hallucinations… that is literally the last gap.

What you are missing here is that the "hallucinations" you don't like and the "results" you do like are, in terms of the underlying process, exactly the same thing. They are not an aberration you can remove. Producing these kinds of results without "hallucinations" is going to require fundamentally different techniques. It's not a "last gap".

Re: Being “Confidently Wrong” is holding AI back

#214

Post by NoGravitas »

ninetyninenine wrote: post #72
sfn42 wrote: post #36

Earlier quoted context omitted.

Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.

No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the ini…

IOW, the realistic position is not held by the majority of people whose paychecks depend on it being wrong. I'm shocked.

Re: Being “Confidently Wrong” is holding AI back

#215

Post by [deleted] »

ninetyninenine wrote: post #189
bigstrat2003 wrote: post #180

Earlier quoted context omitted.

They would have issues even remaining employed. AI defenders are very quick to point out "humans mistakes too", but that is a false equivalence because humans learn . If a junior makes a really stupid mistake, when I show him the correct way he won't make that mistake again. An AI will, because (as people correctly point out) it has no actual intelligence .

There’s examples of humans who can’t learn. Have you seen the movie memento. There are cases where humans lose all ability to form long term memories and outside of a timed context window they remember nothing. That context window is minutes at best. According to your logic these people have no actual intelligence or sentience. Therefore they should be euthanized. You personally can grab a gun and execute each of the…

[deleted]

Re: Being “Confidently Wrong” is holding AI back

#216

Post by Dilettante_ »

energy123 wrote: post #117
rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Re online learning - If I freeze 40 yo Einstein and make it so he can't form new memories beyond 5 minutes, that's still an incredibly useful, generally intelligent thing. Doesn't seem like a problem that needs to be solved on the critical path to AGI. Re training data - We have synthetic data, and we probably haven't hit a wall. Gpt-5 was only 3.5 months after o3. People are reading too much into the tea leaves here…

>If I freeze 40 yo Einstein and make it so he can't form new memories beyond 5 minutes, that's still an incredibly useful, generally intelligent thing.

But every time you tried to get him to do something you'd have to teach him from first principles. Good luck getting ChatStein to interact with the internet, to write code or design a modern airplane. Even in physics, he'd be using antiquated methods and assumptions, this getting worse as time progresses(like sib comment I believe was alluding to).

And don't even get me started on the language barrier.

I recently read this short story[1] on the topic so it's fresh on my mind.

[1]https://qntm.org/mmacevedo

Re: Being “Confidently Wrong” is holding AI back

#217

Post by ofrzeta »

LLMs just can't learn or understand from the context. The context is there to somehow statistically affect the token production but there is no real understanding. You can provide an LLM a full specification of a problem including all elements that are needed to solve it, for instance all specific functions of a programming library (that is not on the Internet). An competent programmer could read this and implement the solution straightforward. With LLMs this does not work - they still confidently continue producing wrong solutions, though.

Re: Being “Confidently Wrong” is holding AI back

#218

Post by Dilettante_ »

procaryote wrote: post #209
rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Also that no companies involved seem to be making a profit, have a reasonable vision to make a profit, or even revenue in the same ballpark as costs. Except nvidia perhaps

But that's not so unusual these days, no? I'm not terribly knowledgeable in the field, but don't Amazon, Netflix, Uber and such work on this kind of funding structure?

Re: Being “Confidently Wrong” is holding AI back

#219

Post by vidarh »

recursive wrote: post #197
vidarh wrote: post #139

Earlier quoted context omitted.

This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is w…

This is also a thing that's bad about LLMs. You're holding it wrong if you continue to argue. But LLMs are presented as if we can use the conventions of natural language to communicate with them. That's how they're sold. So if they fail to live up to those expectations, that's still a problem with LLMs.

It's a problem with LLM's and people are "holding it wrong".

It makes zero difference that they've been sold as doing better if other people learn how to use them effectively and I choose to ignore how to get the best possible results out of them.

Re: Being “Confidently Wrong” is holding AI back

#220

Post by vidarh »

_flux wrote: post #162
vidarh wrote: post #139

Earlier quoted context omitted.

This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is w…

Indeed, arguing with LLM is good if you like arguing. For results it's not the way to go. I think often it's not required to completely start over: just identify the part where it goes off the rails, and modify your prompt just before that point. But yeah, basically the same process.

Sure, when working with tools - like Copilot for example - that lets you "restore" the conversation to a given point, that has pretty much the same effect. The key is to excise the "bad steps" from the conversation and figure out how to amend the next conversation steps so it doesn't veer off in the wrong direction.
Post reply on HN