Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

261–270 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#261

Post by brookst »

kelseyfrog wrote: post #195
brookst wrote: post #188

Earlier quoted context omitted.

You're conflating "correct themselves" with "are guaranteed to give the correct answer", which are two really different things. And in fact you're just echoing GP's point: their corrections can be wrong. You case is no different from: - AI: "The capital of France is Paris" - User: "This is wrong, it changed to Montreal in 2005" - AI: "You're absolutely right! The capital of France is Montreal"

Instead I get this: Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.

I was using an oversimplified example to illustrate. In practice, it appears more in large context statements about the context. If the human is wrong, there’s a good chance the AI will cheerfully agree and then be wrong too.

I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3, just use the existing ModuletExist”. Claude tied itself in knots because it believes me.

(The module does exist in another project I’m working on)

Re: Being “Confidently Wrong” is holding AI back

#262

Post by AlecSchueler »

gloosx wrote: post #254

We had many examples of AIs which tried to learn from feedback in the public domain. They all quickly becoming racist nazis for some reason.

What are examples other than Grok which apparently had nazi sympathies hardcoded in the system prompt?

Re: Being “Confidently Wrong” is holding AI back

#263

Post by AlecSchueler »

JCM9 wrote: post #64

Add to being confidently wrong is the super annoying way it corrects itself after disastrously screwing something up. AI: “I’ve deployed the API data into your app, following best practices and efficient code.” Me: “Nope thats totally wrong and in fact you just wrote the API credential into my code, in plaintext, into the JavaScript which basically guarantees that we’re gonna get hacked.” AI: “You’re absolutely right…

And then proceeds not to fix it.

Re: Being “Confidently Wrong” is holding AI back

#264

Post by ninetyninenine »

dns_snek wrote: post #260
ninetyninenine wrote: post #256

Earlier quoted context omitted.

No there is no confounding. When you hallucinate with schizophrenia you know things that are not true and you sense things that are not true. The hallucinations involve both sensory and knowledge. A weak conceptual model of the world is the problem. But realize humans also have a weak conceptual model of the world as well and make a bunch of hallucinations based on that weak model. For example many people are still m…

> It’s not an incorrect model of the world as technically both you and an LLM ultimately have an incorrect model of the world and both you and the LLM fake it. I should've said that the model is "missing", not "weak" when talking about LLMs, that was my mistake. Yes I'm a human with an imperfect and in many aspects incorrect conceptual model of the world, that is true. The following aren't real examples, they're hype…

No it doesn’t. Conceptual understanding is there. But the LLM is not obligated towards correctness. The fact that at one point it gave you the correct answer is indicative that an aspect of it understands the concept.

Like if I told it solve a complex puzzle equation not in its training data and it correctly solved that problem. We know from the low probability of arriving at that solution from random chance that the LLM must know and understand and reason to arrive at that solution.

Now you’re saying you perturb the input with some grammar changes but leave everything else the same and the LLM will now produce a wrong answer. But this doesn’t change the fact that it was able to get the right answer.

Humans can be dumb and inconsistent. LLMs can be dumb and inconsistent too. This happens to be a quirk of the LLM. But you cannot deny that it is intelligent on the sole fact that LLMs can produce output that we know for sure can only be arrived at through reasoning.

Re: Being “Confidently Wrong” is holding AI back

#265

Post by projektfu »

IsTom wrote: post #145
projektfu wrote: post #29

Earlier quoted context omitted.

The article buries the lede by waiting until the very end to talk about solutions like having the LLM write DSL code. Presumably if you feed an LLM your orders table and a question about it, you'll get an answer that you can't trust. But if you ask it to write some SQL or similar thing based on your database to get the answer and run it, you can have more confidence.

Until it mishandles a NULL somewhere in a condition on does JOIN instead of a LEFT JOIN and outputs something plausibly-looking that is just plain wrong. To verify it you'll need to do the work that it would take to write it anyway.

I disagree, both because LLMs can be less likely to make those errors than a lot of humans, and because it's easier for me to review and critique its code than to review my own. I can also have a basis for testing, and I can tell it to fix problems in the code rather than having it make up a new answer.

If what I am doing is summarizing data and it will likely have uncertainty as a result, I can include statistics in the specification of what I want.

I have also been impressed from time to time where Claude Code catches a mistake I would have written. For example, I asked it to create a configuration file with some names of my staff to use for a query. It then ran the query and noticed that one name I gave was not in the database, but that there was a similar name, and it recommended changing the config.

I am pessimistic about whether these tools are intelligent or will ever achieve intelligence, but where they are useful, we should use them.

Re: Being “Confidently Wrong” is holding AI back

#266

Post by kelseyfrog »

brookst wrote: post #261
kelseyfrog wrote: post #195

Earlier quoted context omitted.

Instead I get this: Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.

I was using an oversimplified example to illustrate. In practice, it appears more in large context statements about the context. If the human is wrong, there’s a good chance the AI will cheerfully agree and then be wrong too. I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3…

Help me understand how it's tangibly different from a veteran telling the rookie to find headlight fluid, winter air for the tires, or keys to the bomb range.

Re: Being “Confidently Wrong” is holding AI back

#267

Post by giancarlostoro »

asadotzler wrote: post #242
giancarlostoro wrote: post #102

What's really funny to me is, sometimes it fixes itself if you just ask "are you SURE ABOUT THIS ANSWER?" myself and others often wonder, why the heck don't they run a 2nd model to "proofread" output or spot check it. Like did you actually answer the question or are you going off a really weird tangent. I asked Perplexity some question for sample UI code for Rust / Slint, it gave me a beautiful web UI, I think it got…

I've asked that question on accurate answers and had the bot say oops and change the answer to an inaccurate one. This seems to happen with about the same frequency on both sides so I'm not sure how helpful it will ultimately be.

Interesting! Have not tried that

Re: Being “Confidently Wrong” is holding AI back

#268

Post by dns_snek »

ninetyninenine wrote: post #264
dns_snek wrote: post #260

Earlier quoted context omitted.

> It’s not an incorrect model of the world as technically both you and an LLM ultimately have an incorrect model of the world and both you and the LLM fake it. I should've said that the model is "missing", not "weak" when talking about LLMs, that was my mistake. Yes I'm a human with an imperfect and in many aspects incorrect conceptual model of the world, that is true. The following aren't real examples, they're hype…

No it doesn’t. Conceptual understanding is there. But the LLM is not obligated towards correctness. The fact that at one point it gave you the correct answer is indicative that an aspect of it understands the concept. Like if I told it solve a complex puzzle equation not in its training data and it correctly solved that problem. We know from the low probability of arriving at that solution from random chance that the…

> The fact that at one point it gave you the correct answer is indicative that an aspect of it understands the concept.

Having a conceptual understanding means that you always provide the same answer to a conceptually equivalent question. Producing the wrong answer when a question is rephrased is indicative of rote memorization.

The fact that it provided the right answer at one point is only indicative of memorization, not understanding which is precisely the difference between sometimes getting it right and always getting it right.

Re: Being “Confidently Wrong” is holding AI back

#269

Post by sfn42 »

dns_snek wrote: post #258
ninetyninenine wrote: post #170

Earlier quoted context omitted.

It doesn’t but I would argue that evidence is in favor of the majority. The ai can easily answer correctly complex questions NOT in its data set. If it is generating answers to questions like these out of thin air which fits our colloquial definition of intelligence.

LLMs also fail to answer simple questions that require a minimal amount of reasoning which demonstrates that they do not have a working model of the world. Their answers to factual questions will change depending on how you phrase the question, even if the crux of the question is identical: "Is X true" -> "Yes, X is true." "Is X a myth?" -> "Yes, X is a myth" "Is Y a myth?" (where X = Y, rephrased) -> "No, Y is true"…

I kind of started this thread but didn't have the energy to argue about it. You provided the exact argument I wanted, thanks for that. This is exactly the reason why I am adamant that LLMs are not intelligence.

They do this really impressive stuff like generate code and hold conversations that makes them seem intelligent, but then they fail at these extremely basic tasks which, to me, proves that it's all an illusion.

It doesn't understand the instructions you give it, it doesn't even understand the answer it gives you. It just consumes and generates tokens. Sure it works pretty well and it's pretty cool stuff, but it's not AI.

Re: Being “Confidently Wrong” is holding AI back

#270

Post by dns_snek »

ninetyninenine wrote: post #259
dns_snek wrote: post #258

Earlier quoted context omitted.

LLMs also fail to answer simple questions that require a minimal amount of reasoning which demonstrates that they do not have a working model of the world. Their answers to factual questions will change depending on how you phrase the question, even if the crux of the question is identical: "Is X true" -> "Yes, X is true." "Is X a myth?" -> "Yes, X is a myth" "Is Y a myth?" (where X = Y, rephrased) -> "No, Y is true"…

So. Humans hallucinate too and get simple shit wrong all the time too. That’s the current problem with LLMs we know it gets shit wrong. We also know humans get shit wrong and make hallucinatory claims but that doesn’t make us classify humans as not intelligent. The fact of the matter is that as retarded and as stupid as the LLM is the fact that it’s so prevalent in the world today is because it gets answers right. We…

So this has nothing to do with "getting things wrong" and everything to do with why they get things wrong.

They get simple, rephrased, but conceptually equivalent questions really wrong and they do this:

1. while the context already contains their previous answer to the original question (which was correct),

2. while the context contains all background information on the topic that would allow an intelligent being to arrive at the correct answer through simple logical deduction,

3. without recognizing or acknowledging that they provided a conflicting answer (without being prompted),

4. while denying that the two answers are contradictory if that fact is pointed out to them,

5. while fabricating a list of bogus reason justifying a different answer if pressed for an explanation.

That's one common failure mode, the other common failure mode is where they uncritically accept our own erroneous corrections even when the correction contain obviously flawed reasoning.

This behavior demonstrates a fundamental lack of conceptual understanding of the world and points at rote memorization in the general case. Maybe LLMs develop a more conceptual understanding of a certain topic when they've been benchmaxxed on that topic? I don't know, I'm not necessarily arguing against that, not today anyway.

But these errors are a daily occurrence in the general case when it comes to any topic they haven't been benchmaxxed for - they certainly don't have a conceptual understanding of cooking, baking, plumbing, heating, electrical circuits, etc.

Post reply on HN