Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Folk are getting dangerously attached to AI that always tells them they're right

theregister.com

Post reply on HN
Print view View original

61–70 of 240 posts

Re: Folk are getting dangerously attached to AI that always tells them they're right

#61

Post by camillomiller »

karmakurtisaani wrote: post #23
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

What I hate even more is when you ask something problematic about another system and they immediately start by reassuring your problem is common and you’re not bad for having the issue. I just need a solution to a normal knowledge issue, why does it always have to assume I’m frustrated already and in need of reassurance?

Re: Folk are getting dangerously attached to AI that always tells them they're right

#62

Post by xenocratus »

joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

> It's just a box of numbers, really cool numbers, with really cool math, that can do really cool things, but still just numbers.

https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...

Re: Folk are getting dangerously attached to AI that always tells them they're right

#63

Post by harvey9 »

Levitz wrote: post #33
cogman10 wrote: post #28

Earlier quoted context omitted.

Let's be serious, it's not like AI companies haven't fed into this misunderstanding. CEOs of these companies love to muse about the possibility that an LLM is conscious.

I presume wasps are conscious. I still don't trust wasps.

Is that how you approach debugging? :)

Re: Folk are getting dangerously attached to AI that always tells them they're right

#64

Post by throwatdem12311 »

joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

My first reaction is to go research it myself. Asking a slop generator yes-man to criticize something for you is still slop.

I pretty much never ask an LLM for a judgment call on anything. Give me facts and references only. I will research and make the judgement myself.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#65

Post by harvey9 »

jsw97 wrote: post #40
yarn_ wrote: post #34

Earlier quoted context omitted.

"It would be astonishing if people were able to casually not antropomorphize LLMs" Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards…

Maybe it is a dangerous habit to instruct entities in plain English without anthropomorphizing them to some extent, without at least being polite? It should feel unnatural do that.

It does feel unnatural to me. I want to be frugal with compute resource but I then have to make sure I still use appropriate language in emails to humans.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#66

Post by zone411 »

I built two related benchmarks this month: https://github.com/lechmazur/sycophancy and https://github.com/lechmazur/persuasion. There are large differences between LLMs. For example, good luck getting Grok to change its view, while Gemini 3.1 Pro will usually disagree with the narrator at first but then change its position very easily when pushed.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#67

Post by roywiggins »

My_Name wrote: post #36

I have the opposite reaction, when it is confident, or says I am right, I accuse it of guessing to see what it says. I say "I think you are getting me to chase a guess, are you guessing?" 90% of the time it says "Yes, honestly I am. Let me think more carefully." That was a copypasta from a chat just this morning

it's mostly just agreeing with you (that yes, it was guessing). LLMs have very limited ability to even know whether it was guessing. But it can "cheat" and just say yes it was if it seems like that's what you expect to hear.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#68

Post by saltcured »

joshstrange wrote: post #25
Sharlin wrote: post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also a…

It's one of those metaphors you cannot even appreciate unless you've been through the technical history.

"It's a collection of warehouses of computers where the system designers gave up on even making a system diagram, instead invoking the cloud clipart to represent amorphous interconnection."

Re: Folk are getting dangerously attached to AI that always tells them they're right

#69

Post by throwatdem12311 »

My_Name wrote: post #36

I have the opposite reaction, when it is confident, or says I am right, I accuse it of guessing to see what it says. I say "I think you are getting me to chase a guess, are you guessing?" 90% of the time it says "Yes, honestly I am. Let me think more carefully." That was a copypasta from a chat just this morning

and it doesn’t actually think more carefully

these things are incapable of thinking, no matter what the UI and marketing calls it

Re: Folk are getting dangerously attached to AI that always tells them they're right

#70

Post by saltcured »

joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I have recently formed an untestable hypothesis, which is that my similar (or stronger) resistance to this comes from having grown up in direct contact with mentally ill family.

In some ways, my theory of mind includes a lot more second guessing as a defense mechanism. At a foundational level, I know there can be hallucination and delusion that leaks out, even when the other party is in peak form and doing their best to mask it and pass as functional.

Post reply on HN