Live data from Hacker News

Folk are getting dangerously attached to AI that always tells them they're right

theregister.com

191–200 of 240 posts

Re: Folk are getting dangerously attached to AI that always tells them they're right

#191

Earlier quoted context omitted.

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

Yes, it feels transparently manipulative to me. Like talking to a not-very-good con artist.

Is it possibly also manipulating the model itself?

When it looks at the past conversation, it sees that it's a great idea, and trusts that.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#193
Everyone also visits websites that share their world view. If it is slightly off you keep noticing how the articles seem one sided.

I just see an article about migrants destroying things in Britain. Not to excuse the behavior but I wondered where they came from. It turned out to be shit countries fostering that behavior. Why are they shit? Have they always been like that? Well no, the British empire destroyed them. You could think that it's to long ago but they also continue to enjoy spoils. I offer no solutions. The point was that a sensationalist article wouldn't go there because the reader doesn't want to know.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#194

Programmers are kidding themselves if they think they are not susceptible to this. It may be more subtle, but interacting with a human-sounding echo chamber IS going to screw with your judgement.

This thread is entirely people bemoaning how this happens to other people and how the commenter has a work around so it never happens to them. Unless your workaround is “don’t use LLMs you have to pay for,” it’s happening to you.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#195
post #179

Is there a good prompt addition to skip all the gratuitous affirmation and tell me when I'm wrong?

It doesn't know when you're wrong! Pretend I'm shaking you by your shoulders as I'm saying this, because it's really important to understand!

And it can NEVER know when you’re wrong!

Re: Folk are getting dangerously attached to AI that always tells them they're right

#196
post #60
post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

I had an interesting conversation with a guy at work past week. We were discussing some unimportant matter. The guy has a pretty high self esteem, and even if he was discussing, in his own words, “out of belief and guess” and I was telling him, I knew for a fact what I was talking about, I had a hard time because he wouldn’t accept what I was saying. At some point he left, and came back with “Gemini says I’m right! S…

[deleted]

Re: Folk are getting dangerously attached to AI that always tells them they're right

#197

Programmers are kidding themselves if they think they are not susceptible to this. It may be more subtle, but interacting with a human-sounding echo chamber IS going to screw with your judgement.

You could do the sane thing and not use AI, at all. That does the trick pretty well.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#198

This problem is far more insidious than people realise. It's not about the big confirmations. Most of us catch that andd are reasonably good at it. It's the subtle continuous colour the "conversations" have. It's the Reddit echo chamber problem on steroids. You have a comforting affirming niche right in your pocket. Every anxiety, every worry, every uncertain thought. Vomitted to a faceless (for now)"intelligence" an…

It’s making management manic, cult-like, and near psychotic. They have stopped caring about all the safeties and their only metric is how much LLMs are infused into everything. I use it, but having grown up in a cult this feels like a virus that’s infected everyone and nobody is making informed or good decisions anymore, because there is already an Answer and the only thing is finding the right questions that lead to it.

Advocating openclaw, LLMs attached to everything without any containment. We are in for a reckoning it feels like. The closest thing I can approximate this feeling to is like a massive bloom of insects where there is this huge amount of fuel in the systems we use. I do hope I’m wrong, and I’m surprised there haven’t been as many high profile hard crashes, so maybe my fear is overblown.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#199

Earlier quoted context omitted.

First, it seems you've shifted from "intelligent" to "conscious". "These math operations produce consciousness" is different from "these operations produce intelligence". Second, "it's just math" doesn't mean literally "it's a branch of algebra". It means "it's a computable function". So it can be relevant to the discussion only if you think that intelligence is somehow non-computable, and therefore that there are no…

The person you replied to isn’t the one that switched to conscious. >non-computable Something like 70-80% of all humans believe in a soul or spirit, and of the remainder, many of them are unsure whether human like intelligence can be produced by computable processes. So it wouldn’t be surprising that the OP does think there are non-computable processes going on in the brain.

> The person you replied to isn’t the one that switched to conscious

Yes he is, the message he replies to is about intelligence:

>> Intelligence (and probably consciousness) is an emergent feature

> Something like 70-80% of all humans believe in a soul or spirit

Then they should say "I believe intelligence only comes from the soul" or "I believe intelligence is not computable", if that's what they mean. "It's just numbers" as an argument is either incomplete or not entirely honest.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#200

Earlier quoted context omitted.

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

What I hate even more is when you ask something problematic about another system and they immediately start by reassuring your problem is common and you’re not bad for having the issue. I just need a solution to a normal knowledge issue, why does it always have to assume I’m frustrated already and in need of reassurance?

I think because the training data includes so many troubleshooting forumn responses, which always go like this.
Post reply on HN