Live data from Hacker News

Folk are getting dangerously attached to AI that always tells them they're right

theregister.com

141–150 of 240 posts

Re: Folk are getting dangerously attached to AI that always tells them they're right

#141

Earlier quoted context omitted.

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

That's the part most people miss—and here's why it actually matters. That signal is real, and it’s hard to ignore.

BINGO, now I know exactly what the problem is.

I've fixed the issue and the code is now fully verified and production ready.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#142

Earlier quoted context omitted.

Although I do think they're not conscious (yet). I think the reasoning 'it's just math' doesn't hold up. Intelligence (and probably consciousness) is an emergent feature of any sufficiently complex network of learning/communicating/selforganizing nodes (that is benefited by intelligence). I don't think it really matters whether it's implemented in math, mycelium, by ants in a hive or in neurons.

The "it's just math" argument may not be technically rigorous, but it's directionally correct. The unstated reasoning invites us to consider why this particular math would be conscious, but not many other forms of math all around us.

First, it seems you've shifted from "intelligent" to "conscious". "These math operations produce consciousness" is different from "these operations produce intelligence".

Second, "it's just math" doesn't mean literally "it's a branch of algebra". It means "it's a computable function". So it can be relevant to the discussion only if you think that intelligence is somehow non-computable, and therefore that there are non-computable processes going on in our brain. Otherwise it's a perfectly pointless remark.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#143
post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

> Nontechnical people simply don't have any idea about what LLMs are. We're on HN, a highly technical corner of the internet, yet we see the same stuff here. It's not just non-technical people... I think one of the big dangers is that people (including us) are quick to believe "I'm better than that". Yet this is a bias conmen have been exploiting for millennia. The only real defense is not lulling yourself into a fal…

There’s nobody who knows how to fool you better than yourself.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#144

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

This drives me nuts. "What a clever question to ask! You must be one of the brightest minds of your generation. Nothing slips by you. Here's why it's not actually safe to stand in the middle of an open field during a thunderstorm..."

Re: Folk are getting dangerously attached to AI that always tells them they're right

#145
post #34
post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

"It would be astonishing if people were able to casually not antropomorphize LLMs" Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards…

  > Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards the model, etc.
Worse, models often perform better when using that natural language because that's what kind of language they were trained on. I say worse because by speaking that way to them you will also naturally humanize them too.

(As a ml researcher) I think one of the biggest problems we have is that we're trying to make a duck by making an animatronic duck indistinguishable from a real duck. In some sense this makes a lot of sense but it also only allows us to build a thing that's indistinguishable from a real duck to us, not indistinguishable from a real duck to something/someone else. It seems like a fine point, but the duck test only allows us to conclude something is probably a duck, not that it is a duck.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#147

Earlier quoted context omitted.

That's the part most people miss—and here's why it actually matters. That signal is real, and it’s hard to ignore.

*twitch* I also like when it says "this is a known issue!" to try and get out of debugging and I ask for a link and it goes "uh yeah I made that up".

Right, because in the training set, text like that is often followed by the text “this is a known issue!”.

That’s a great example to use to explain to people why these things are not actually reasoning.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#148
post #65

Earlier quoted context omitted.

It does feel unnatural to me. I want to be frugal with compute resource but I then have to make sure I still use appropriate language in emails to humans.

This. Right now, I'm assuming you're all humans, and so are all my coworkers, and the other people driving cars around me and etc. How easy is it to dehumanize actual humans? If I don't try to remain polite in all written English conversations, including the LLMs, that's going to trickle over to the rest of my interactions too. Doesn't mean they deserve it, just that it's a habit I know I need to maintain.

You're only polite out of habit?

Re: Folk are getting dangerously attached to AI that always tells them they're right

#149
This problem is far more insidious than people realise.

It's not about the big confirmations. Most of us catch that andd are reasonably good at it.

It's the subtle continuous colour the "conversations" have.

It's the Reddit echo chamber problem on steroids.

You have a comforting affirming niche right in your pocket.

Every anxiety, every worry, every uncertain thought.

Vomitted to a faceless (for now)"intelligence" and regurgitated with an air of certainty.

Will people have time to ponder at all going forwards?

Re: Folk are getting dangerously attached to AI that always tells them they're right

#150
post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also a…

The Cloud is a just a computer that you don’t own, located in Reston, Virginia.
Post reply on HN