Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI overly affirms users asking for personal advice

news.stanford.edu

Post reply on HN
Print view View original

301–310 of 684 posts

Re: AI overly affirms users asking for personal advice

#301

Post by JoshTriplett »

bloomca wrote: post #261
astrange wrote: post #246

Earlier quoted context omitted.

Claudes have lots of empathy. The issue is the opposite - it isn't very good at challenging you and it's not capable of independently verifying you're not bullshitting it or lying about your own situation. But it's better than talking to yourself or an abuser!

It's about the same as talking to yourself, LLMs simply agree with anything you say unless it is directly harmful. Definitely agree about talking to an abuser, though. Sometimes people indeed just need validation and it helps them a lot, in that case LLMs can work. Alternatively, I assume some people just put the whole situation into words and that alone helps. But if someone needs something else, they can be straigh…

> It's about the same as talking to yourself

In one way it's potentially worse than talking to yourself. Some part of you might recognize that you need to talk to someone other than yourself; an LLM might make you feel like you've done that, while reinforcing whatever you think rather than breaking you out of patterns.

Also, LLMs can have more resources and do some "creative" enabling of a person stuck in a loop, so if you are thinking dangerous things but lack the wherewithal to put them into action, an LLM could make you more dangerous (to yourself or to others).

Re: AI overly affirms users asking for personal advice

#302

Post by worksonmine »

nuancebydefault wrote: post #294
shimman wrote: post #173

Earlier quoted context omitted.

You can't be careful at all doing this, this is like smoking a cigarette in a dynamite factory. Using LLMs for therapy is so deeply dystopian and disgusting, people need human empathy for therapy. LLMs do not emit empathy. Complete disaster waiting to happen for that individual.

My experience is that it tries to look at your situation in an objective way, and tries to help you to analyse your thoughts and actions. It comes across as very empathetic though, so there can lie a danger if you are easily persuaded into seeing it as a friend.

It doesn't try to do anything. It doesn't work like that. It regurgitates the most likely tokens found in the training set.

Re: AI overly affirms users asking for personal advice

#303

Post by catlifeonmars »

epistasis wrote: post #265
zulban wrote: post #248

Earlier quoted context omitted.

Generally, published papers don't give a damn about reproducibility. I've seen it identified as a crisis by many. Publishers, reviewers, and researchers mostly don't care about that level of basic rigor. There's no professional repercussions or embarrassment. Agreed - if I was a reviewer for LLM papers it would be an instant rejection not listing the versions and prompts used.

I'm not so sure of that opinion on reproducibility. The last peer review I did was for a small journal that explicitly does not evaluate for high scientific significance, merely for correctness, which generally means straightforward acceptance. The other two reviews were positive, as was mine, except I said that the methods need to be described more and ideally the code placed somewhere. That was enough for a complet…

> and instead focus on the results...

This points to (and everyone knows this) incentives misalignment between the funders of research and the public. Researchers are caught in the middle

Re: AI overly affirms users asking for personal advice

#304

Post by ghywertelling »

zulban wrote: post #248
anorwell wrote: post #232

A pastime I have with papers like this is to look for the part in the paper where they say which models they tested. Very often, you find either A) it's a model from one or more years ago, only just being published now, or B) they don't even say which model they are using. Best I could find in this paper: > We evaluated 11 user-facing production LLMs: four proprietary models from OpenAI, Anthropic, and Google; and se…

Generally, published papers don't give a damn about reproducibility. I've seen it identified as a crisis by many. Publishers, reviewers, and researchers mostly don't care about that level of basic rigor. There's no professional repercussions or embarrassment. Agreed - if I was a reviewer for LLM papers it would be an instant rejection not listing the versions and prompts used.

The same about surveys and polls. I know no one who has ever been polled or surveyed. When will we stop this fascination with made up infographics crisis?

Re: AI overly affirms users asking for personal advice

#305

Post by rustystump »

gjm11 wrote: post #197
plagiarist wrote: post #111

Earlier quoted context omitted.

People are upset hearing that LLMs aren't sentient for some reason. Expect to be downvoted, it is okay.

First off, "not adequately described as a mere token-predictor" and "not sentient" are entirely separate things. I can't speak for anyone else, but what I feel when I read yet another glib "it's just a stochastic parrot, of course it isn't doing anything that deserves to be called reasoning" take is much more like bored than it is like upset . Today's LLMs are in some sense "just predicting tokens" in some sense. Lik…

I wont touch how profoundly i disagree with everything you said on reasoning (u clearly already have it figured out) but a fun test i have done with most of the big models is to give it some text input, maybe a short story, and have it rate it. That is, the prompt is, rate this from 1-10.

For Gemini and gpt, it almost always will give very similar scores for everything. As long as grammar isnt off u cannot get below a 7.

X ai on the other hand will rarely give anything above a 7.

Now when u prompt with, rate 1-10 with 5 being average, all the sudden the scores of openai and gemini drop and x ai remains roughly the same.

All of them will eventually give you a 10 if u keep making tiny edits “fixing” whatever they complain about.

Humans do not do this. Or more specifically, my experience with humans.

Re: AI overly affirms users asking for personal advice

#306

Post by user_7832 »

NortySpock wrote: post #216
dimgl wrote: post #124

Even as someone who (wrongly) believed that I had high emotional intelligence, I too was bit by this. Almost a year ago when LLMs were starting to become more ubiquitous and powerful I discussed a big life/professional decision with an LLM over the course of many months. I took its recommendation. Ultimately it turned out to be the wrong decision. Thankfully it was recoverable, but it really sobered me up on LLMs. Th…

One mental model I have with LLMs is that they have been the subject of extreme evolutionary selection forces that are entirely the result of human preferences. Any LLM not sufficiently likable and helpful in the first two minutes was deleted or not further iterated on, or had so much retraining (sorry, "backpropagation") it's not the same as it started out. So it's going to say whatever it "thinks" you want it to sa…

Fully agree. I wonder in the long term how this will show up. Will every business/CEO do more of what he/they anyway want to do, but now supported by AI/LLMs?

The possibilities in "dangerous" fields are a bit more frightening. A general is much more likely to ask ChatGPT "Do you think this war is a good idea/should I drop a bomb", rather than an actually helpful tool - where you might ask "What are 5 hidden points on favor of/against bombing that one likely has missed".

The more you use AI as a strict tool that can be wrong, the safer. Unfortunately I'm not sure if that helps if the guy bombing your city (or even your president) is using AI poorly, and their decisions affect you.

Re: AI overly affirms users asking for personal advice

#307

Post by megous »

ranger_danger wrote: post #65
megous wrote: post #28

Can't you just prompt for a critical take, multiple alternative perspectives (specifically not yours, after describing your own), etc.? It's a tool, I can bang my hand on purpose with a hammer, too.

Yes, if you're smart. But most people asking it random questions and expecting it to read their minds and spit out the perfect answer are not so much. They don't know what a prompt is, and wouldn't be bothered to give it prior instructions either way.

Educated, not smart. This is a job for schools to include AI education into the basic curricula. Their pupils will use the tools anyway, so at least teach them to do it with proper expectations and prompting techniques/pitfalls.

Re: AI overly affirms users asking for personal advice

#308

Post by adamtaylor_13 »

Interestingly, you can simply tell models to not be sycophantic and they'll listen.

Claude is almost annoyingly good at pushing back on suggestions because my global CLAUDE.md file says to do so. I rarely get Claude "you're absolutely right"ing me because I tell it to push back.

Re: AI overly affirms users asking for personal advice

#309

Post by yacin »

anorwell wrote: post #232

A pastime I have with papers like this is to look for the part in the paper where they say which models they tested. Very often, you find either A) it's a model from one or more years ago, only just being published now, or B) they don't even say which model they are using. Best I could find in this paper: > We evaluated 11 user-facing production LLMs: four proprietary models from OpenAI, Anthropic, and Google; and se…

Any paper like this would easily take a year or more to write and go through the submission/review/rebuttal/revision/acceptance process. I don't understand why the models being a year or two old now is worth noting as though it's a clear weakness? What should they do, publish sub-standard results more quickly?

Re: AI overly affirms users asking for personal advice

#310

Post by matwood »

CoffeeOnWrite wrote: post #269
salawat wrote: post #263

Earlier quoted context omitted.

>Obviously subservient people default to being yes-men because of the power structure. No one wants to question the boss too strongly. This drives me nuts as a leader. There are times where yes, please just listen, and if this is one of those times, I'll likely tell you, but goddamnit, speak up. If for no other reason I might not have thought of what you've got to say. Then again, I also understand most boss types ar…

Indeed. I directly ask my reports to discover and surface conflicts, especially disagreements with me, and when they do I try to strongly reinforce the behavior by commending and rewarding them. Could anyone recommend additional resources on this topic?

Simon Sinek has a lot of good content around this. Step one is building trust. People won’t speak up if they don’t feel safe doing so.
Post reply on HN