Live data from Hacker News

AI overly affirms users asking for personal advice

news.stanford.edu

141–150 of 684 posts

Re: AI overly affirms users asking for personal advice

#141
post #92
post #83

You're essentially summoning a character to role-play with. Just like with esoteric evocation, it's very easy to summon the wrong aspect of the spirit. Anthropic has a lot to say about this: https://www.anthropic.com/research/persona-selection-model https://www.anthropic.com/research/assistant-axis https://www.anthropic.com/research/persona-vectors

Unfortunately (after reading your links) all of the control surfaces for mitigating spirit summoning seem to be in the model training, creation and tuning not something you can change meaningfully through prompting. Perhaps the LLM itself, rather than the role model you created in one particular chat conversation or another, is better understood to be the “spirit.” As a non-coder who only chats with pre existing LLMs…

I present you

NVIDIA Nemotron-Personas-USA — 1 million synthetic Americans whose demographics match real US census distributions

https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA

Re: AI overly affirms users asking for personal advice

#142
post #14

Maybe it's not so sensible to offload the responsibility of clear thinking to AI companies? How is a chatbot supposed to determine when a user fools even themselves about what they have experienced? What 'tough love' can be given to one who, having been so unreasonable throughout their lives - as to always invite scorn and retort from all humans alike - is happy to interpret engagement at all as a sign of approval?

It's almost as if being a therapist is an actual job that takes years of training and experience! AI may one day rewrite Windows but it will never be counselor Troi.

Implying that programming is not an actual job that takes years of training and experience

To be clear I don't think the AI can do either job

Re: AI overly affirms users asking for personal advice

#146

Gemini is like a devil in this sense - i asked a relationship advice and it just bounced pretty nasty stuff.

Yeah out of curiosity I asked ChatGPT a question about a personal situation and its reply was absolutely scorched-earth mode, telling me to get a lawyer etc over what was almost nothing.

Ah, all the Reddit posts are really showing up from the training data, I see.

Re: AI overly affirms users asking for personal advice

#147
Relevant article from The Atlantic a couple weeks ago, "Friendship, On Demand": https://www.theatlantic.com/family/2026/03/ai-friendship-cha... (gift link)

>The way that generative AI tends to be trained, experts told me, is focused on the individual user and the short term. In one-on-one interactions, humans rate the AI’s responses based on what they prefer, and “humans are not immune to flattery,” as Hansen put it. But designing AI around what users find pleasing in a brief interaction ignores the context many people will use it in: an ongoing exchange. Long-term relationships are about more than seeking just momentary pleasure—they require compromise, effort, and, sometimes, telling hard truths. AI also deals with each user in isolation, ignorant of the broader social web that every person is a part of, which makes a friendship with it more individualistic than one with a human who can converse in a group with you and see you interact with others out in the world.

I also thought this bit was interesting, relative to the way that friendship advice from Reddit and elsewhere has been trending towards self-centeredness (discussed elsewhere in this thread):

>Friendship is particularly vulnerable to the alienating force of hyper-individualism. It is the most voluntary relationship, held together primarily by choice rather than by blood or law. So as people have withdrawn from relationships in favor of time alone, friendship has taken the biggest hit. The idea of obligation, of sacrificing your own interests for the sake of a relationship, tends to be less common in friendship than it is among family or between romantic partners. The extreme ways in which some people talk about friendship these days imply that you should ask not what you can do for your friendship, but rather what your friendship can do for you. Creators on TikTok sing the praises of “low maintenance friendships.” Popular advice in articles, on social media, or even from therapists suggests that if a friendship isn’t “serving you” anymore, then you should end it. “A lot of people are like I want friends, but I want them on my terms,” William Chopik, who runs the Close Relationships Lab at Michigan State University, told me. “There is this weird selfishness about some ways that people make friends.”

Re: AI overly affirms users asking for personal advice

#148
post #41

WTF is "yes-men"? Orignal title: AI overly affirms users asking for personal advice Dear mods, can we keep the title neutral please instead of enforcing gender bias?

Gender bias? I could understand if you felt the title was more provocative in signaling sycophancy but what gender bias? I'm confused. Is this some kind of California thing?

Re: AI overly affirms users asking for personal advice

#149

It feels like I'm fighting uphill battle when it comes to bouncing ideas off of a model. I'll set things up in the context with instructions similar to. "Help me refine my ideas, challenge, push back, and don't just be agreeable." It works for a bit but eventually the conversation creeps back into complacency and syncophancy. I'll check it too by asking "are you just placating me?" the funny thing is that often it'll…

> I'll check it too by asking "are you just placating me?" the funny thing is that often it'll admit that, yes, it wasn't being very critical, and then procede to over correct and become a complete contrarian. and not in a way that's useful either.

It's not admitting anything. Your question diverts it down a path where it acts the part of a former sycophant who is now being critical, because that question is now upstream of its current state.

Never make the mistake of asking an LLM about its intentions. It doesn't have any intentions, but your question will alter its behaviour.

Re: AI overly affirms users asking for personal advice

#150
post #124

Even as someone who (wrongly) believed that I had high emotional intelligence, I too was bit by this. Almost a year ago when LLMs were starting to become more ubiquitous and powerful I discussed a big life/professional decision with an LLM over the course of many months. I took its recommendation. Ultimately it turned out to be the wrong decision. Thankfully it was recoverable, but it really sobered me up on LLMs. Th…

I largely agree, I also thought I was smart enough not to be deluded into a false sense of security, but interacting with an LLM is so tricky and slippery that, more often than not you are forced to believe you just solve a problem no one had solve in a hundred years.

My guideline now for interacting with LLM is only to believe the result if it is factual and easily testable, or if I'm a domain expert. Anything else especially if I'm in complete ignorance about the subject is to approach with a high degree of suspicion that I can be led astray by its sycophancy.

Post reply on HN