Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

201–210 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#201

Post by ausbah »

selfmodruntime wrote: post #49
rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Never before did we have a combination of well and poison where the pollution of the first was both as instantaneous and as easily achieved. I‘ve yet to see a convincing article for artificial training data.

yeah like another commenter said, if you can get synthetic data with some some sort of easily verifiable grounding (math, games, code) models can do very well. this is one of the underpinnings of reinforcement learning that has helped some advancements in past year or so (AFAIK)

Re: Being “Confidently Wrong” is holding AI back

#202

Post by 1vuio0pswjnm7 »

What is interesting IMO about the "confidently wrong" phenomenon is that this was also commonly found in internet forums and online commentary in general prior to widespread use of today's confidently wrong "AI". That is, online commenters routinely were and still are "confidently wrong". IMHO and IME, the "confidentlay wrong" phenonmenon was and still is greater represented in online commnentary than "IRL".

No surprise IMO that, generally, online commenters and so-called "tech" companies who tend to be overly fixated on computers as the solution to all problems, are also the most numerous promoters of confidently wrong "AI".

The nature of the medium itself and those so-called "tech" companies that have sought to dominate it through intermediation and "ad services"^1 could have something to do with the acceptance and promotion of confidently wrong "AI". Namely, its ability to reduce critical thinking and the relative ease with which uninformed opinions, misinformation, and other non-factual "confidently wrong" information can be spread by virtually anyone.

1. If "confidently wrong" information is popular, if it "goes viral", then with few exceptions it will be promoted by these companies to drive traffic and increase ad services revenue.

Please note: I could be wrong.

Re: Being “Confidently Wrong” is holding AI back

#203

Post by tliltocatl »

energy123 wrote: post #117
rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Re online learning - If I freeze 40 yo Einstein and make it so he can't form new memories beyond 5 minutes, that's still an incredibly useful, generally intelligent thing. Doesn't seem like a problem that needs to be solved on the critical path to AGI. Re training data - We have synthetic data, and we probably haven't hit a wall. Gpt-5 was only 3.5 months after o3. People are reading too much into the tea leaves here…

> If I freeze 40 yo Einstein and make it so he can't form new memories beyond 5 minutes, that's still an incredibly useful, generally intelligent thing

A "frozen mind" feels like something not unlike a book - useful, but only with a smart enough "human user", and even so be progressively less useful as time passes.

>Doesn't seem like a problem that needs to be solved on the critical path to AGI.

It definitely is one. I know we are running into definitions, but being able to form novel behavior patterns based on experience is pretty much the essence of what intelligence is. That doesn't necessary mean that a "frozen mind" will be useless, but it would certainly not qualify as AGI.

>We don't have visibility into the IMO/IOI medal winning models.

There are lies, damn lies and LLM benchmarks. IMO/IOI is not necessarily indicative of any useful tasks.

Re: Being “Confidently Wrong” is holding AI back

#204

Post by burnte »

lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

I'm just so glad people are seeing this. I started saying this literally days after ChatGPT came out and I started examining the technology. It's SUPER useful, but it's assistive, it can't be trusted to do things autonomously yet. That's ok, though, it can make human workers more productive, rather than worrying about replacing humans.

Re: Being “Confidently Wrong” is holding AI back

#205

Post by procaryote »

tango12 wrote: post #138
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

The AI being wrong problem is probably not insurmountable. Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed. Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not. If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to h…

Depending on your definitions, either:

* there are already lots of "reasoning" models trying meta-cognition, while still getting simple things wrong

or:

* the models aren't doing cognition, so meta-cognition seems very far away

Re: Being “Confidently Wrong” is holding AI back

#206

Post by jqpabc123 »

ctoth wrote: post #192
jqpabc123 wrote: post #94

Earlier quoted context omitted.

there's something fundamentally flawed the whole field has yet to properly pinpointing and fix. Isn't it obvious? It's all built around probability and statistics. This is not how you reach definitive answers. Maybe the results make sense and maybe they're just nice sounding BS. You guess which one is the case. The real catch --- if you know enough to spot the BS, you probably didn't need to ask the question in the f…

> It's all built around probability and statistics. Yes, the world is probabilistic. > This is not how you reach definitive answers. Do go on? This is the only way to build anything approximating certainty in our world. Do you think that ... answers just exist? What type of weird deterministic video game world do you live in where this is not the case?

How many "r's" are in the word "strawberry"?

I'm certain this simple question has a definitive answer.

Re: Being “Confidently Wrong” is holding AI back

#207

Post by wila »

kilpikaarna wrote: post #121
ninetyninenine wrote: post #72

Earlier quoted context omitted.

No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the ini…

> It is clear to everyone who builds LLMs that the AI is intelligent. So presumably we have a solid, generally-agreed-upon definition on intelligence now? > autocompleting things with humanity changing intelligent content. What does this even mean?

that you're arguing with an LLM :)

Re: Being “Confidently Wrong” is holding AI back

#208

Post by corytheboyd »

gus_massa wrote: post #147
corytheboyd wrote: post #127

Isn’t it obvious that the confidently wrong problem will never go away because all of this is effectively built on a statistical next token matcher? Yeah sure you can throw on hacks like RAG, more context window, but it’s still built on the same foundation. It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obv…

It's easy to solve if they modify they training to remove some weight from Stack Overflow and add more weight to Yahoo! Answers :) . I remember a few years ago, we were planing to make some kind of math forum for students in the first year of the university. My opinion was that it was too easy to do it wrong. On one way you can be like Math Overflow were all the questions are too technical (for first year of the univ…

> For the AI, you want to give it some room to generalize/bullshit.

Totally! In my mind I’ve been playing with the phrase: it’s good at _fuzzy_ things. For example IMO voice synthesis before and after this wave of AI hype is actually night and day! In part, to my fuzzy idea, because voice synthesis isn’t factual, it’s billions of little data points coming together to emulate sound waves, which is incredibly fuzzy. Versus code, which is pointy: it has one/few correct forms, and infinite/many incorrect forms.

Re: Being “Confidently Wrong” is holding AI back

#209

Post by procaryote »

rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Also that no companies involved seem to be making a profit, have a reasonable vision to make a profit, or even revenue in the same ballpark as costs.

Except nvidia perhaps

Re: Being “Confidently Wrong” is holding AI back

#210

Post by procaryote »

jqpabc123 wrote: post #206
ctoth wrote: post #192

Earlier quoted context omitted.

> It's all built around probability and statistics. Yes, the world is probabilistic. > This is not how you reach definitive answers. Do go on? This is the only way to build anything approximating certainty in our world. Do you think that ... answers just exist? What type of weird deterministic video game world do you live in where this is not the case?

How many "r's" are in the word "strawberry"? I'm certain this simple question has a definitive answer.

which apparently hard to come up with by compiling lots of text into a statistical model for what text is most likely to come after your question
Post reply on HN