Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI overly affirms users asking for personal advice

news.stanford.edu

Post reply on HN
Print view View original

331–340 of 684 posts

Re: AI overly affirms users asking for personal advice

#331

Post by cruffle_duffle »

worksonmine wrote: post #302
nuancebydefault wrote: post #294

Earlier quoted context omitted.

My experience is that it tries to look at your situation in an objective way, and tries to help you to analyse your thoughts and actions. It comes across as very empathetic though, so there can lie a danger if you are easily persuaded into seeing it as a friend.

It doesn't try to do anything. It doesn't work like that. It regurgitates the most likely tokens found in the training set.

That is so reductive of an analysis that it is almost worthless. Technically true, but very unhelpful in terms of using an LLM.

It is a first principle though so it helps to “stir the context windows pot” by having it pull in research and other shit on the web that will help ground it and not just tell you exactly what you prompt it to say.

Re: AI overly affirms users asking for personal advice

#332

Post by zulban »

inetknght wrote: post #326
zulban wrote: post #248

Earlier quoted context omitted.

Generally, published papers don't give a damn about reproducibility. I've seen it identified as a crisis by many. Publishers, reviewers, and researchers mostly don't care about that level of basic rigor. There's no professional repercussions or embarrassment. Agreed - if I was a reviewer for LLM papers it would be an instant rejection not listing the versions and prompts used.

> Generally, published papers don't give a damn about reproducibility While this is sadly true, it's especially true when talking about things that are stochastic in nature. LLMs outputs, for example, are notoriously unreproducible.

> LLMs outputs, for example, are notoriously unreproducible.

Only in the same way that an individual in a medical study cannot be "reproduced" for the next study. However the overall statistical outcomes of studying a specific LLM can be reproduced.

Re: AI overly affirms users asking for personal advice

#333

Post by zulban »

epistasis wrote: post #265
zulban wrote: post #248

Earlier quoted context omitted.

Generally, published papers don't give a damn about reproducibility. I've seen it identified as a crisis by many. Publishers, reviewers, and researchers mostly don't care about that level of basic rigor. There's no professional repercussions or embarrassment. Agreed - if I was a reviewer for LLM papers it would be an instant rejection not listing the versions and prompts used.

I'm not so sure of that opinion on reproducibility. The last peer review I did was for a small journal that explicitly does not evaluate for high scientific significance, merely for correctness, which generally means straightforward acceptance. The other two reviews were positive, as was mine, except I said that the methods need to be described more and ideally the code placed somewhere. That was enough for a complet…

I'm not sure how one example contradicts documented huge overall trends, but okay.

Re: AI overly affirms users asking for personal advice

#334

Post by SJMG »

legacynl wrote: post #249
trimbo wrote: post #183

> They also included 2,000 prompts based on posts from the Reddit community r/AmITheAsshole, where the consensus of Redditors was that the poster was indeed in the wrong. Sorry, anonymous people on reddit aren't a good comparison. This needs to be studied against people in real life who have a social contract of some sort, because that's what the LLM is imitating, and that's who most people would go to otherwise. Obv…

> Sorry, anonymous people on reddit aren't a good comparison. Yeah especially on r/AmITheAsshole. Those comments never advocate for communication, forgiveness and mending things with family.

Yes, it is a toxic sub, where the notion that there can be greater happiness on the other side of forgiveness than cutting ties is all but absent.

Re: AI overly affirms users asking for personal advice

#335

Post by yacin »

anorwell wrote: post #318
yacin wrote: post #309

Earlier quoted context omitted.

Any paper like this would easily take a year or more to write and go through the submission/review/rebuttal/revision/acceptance process. I don't understand why the models being a year or two old now is worth noting as though it's a clear weakness? What should they do, publish sub-standard results more quickly?

> I don't understand why the models being a year or two old now is worth noting as though it's a clear weakness? I do think it's a clear weakness. Capabilities are extremely different than they were twelve months ago. > What should they do, publish sub-standard results more quickly? Ideally, publish quality results more quickly. I'm quite open to competing viewpoints here, but it's my impression that academic publish…

The onus is on you to prove or at least convincingly argue that the results are unlikely to generalize across incremental model releases. In my personal experience, the overly affirming nature seems to have held since GPT-3. What makes you think a newer, larger model would not exhibit this behavior? Beyond "they're more capable"? I'd argue that being more capable doesn't mean less sycophantic.

It's certainly possible some of the new advances (chain-of-thought, some kind of agentic architecture) could lessen or remove this effect. But that's not what the paper was studying! And if you feel strongly about it, you could try to further the discussion with results instead of handwavingly dismissing others' work.

Re: AI overly affirms users asking for personal advice

#336

Post by epistasis »

catlifeonmars wrote: post #303
epistasis wrote: post #265

Earlier quoted context omitted.

I'm not so sure of that opinion on reproducibility. The last peer review I did was for a small journal that explicitly does not evaluate for high scientific significance, merely for correctness, which generally means straightforward acceptance. The other two reviews were positive, as was mine, except I said that the methods need to be described more and ideally the code placed somewhere. That was enough for a complet…

> and instead focus on the results... This points to (and everyone knows this) incentives misalignment between the funders of research and the public. Researchers are caught in the middle

Eh, I'm not so sure about the funding side there, researchers are not really caught at all and are fully responsible, IMHO. Peer reviewers exist to enforce community standards, and are not influenced to avoid reproducibility concerns by funding sources. The results are always more interesting than reproducibility, of course, and I think that's why the get the attention! Also, there needs to be greater involvement of grad students (who do most of the actual work) in peer review, IMHO, because most PIs spend their day in meetings reviewing results, setting directions, writing grants, and have little time for actual lab work, and are thus disconnected from it.

There needs to be more public naming and shaming in science social media and in conference talks, but especially when there are social gatherings at conferences and people are able to gossip. There was a bit of this with Google's various papers, as they got away with figurative murder on lack of reproducibility for commercial purposes. But eventually Google did share more.

Most journals have standards for depositing expensive datasets, but that's a clear yes/no answer. Reproducibility is a very subjective question in comparison to data deposition, and must be subjectively evaluated by peer reviewers. I'd like to see more peer review guidelines with explicit check boxes for various aspects of reproducibility.

Re: AI overly affirms users asking for personal advice

#337

Post by xiphias2 »

michaelcampbell wrote: post #21
xiphias2 wrote: post #9

Marc Andereseen has talked about the downside of RLHF: it's a specific group of liberal low income people in California who did the rating, so AI has been leaning their culture. I think OpenAI tried to diversify at least the location of the raters somewhat, but it's hard to diversify on every level.

Do you have any links to documentation of this? Andreesen has a definite bias as well, so I'm not about to just accept his say-so in a fit of Appeal to Authority. (eg: "Cite?")

He was talking about it in the Lex Friedman interview after Trump was elected. And he was talking about a lot of things the Biden administration forced on Silicon Valley at that time (since then Google lost a case about one of these back-deals).

Re: AI overly affirms users asking for personal advice

#338

Post by dimgl »

xXSLAYERXx wrote: post #320
layla5alive wrote: post #133

Earlier quoted context omitted.

Any more context you're willing to share?

We really do love dirty laundry don't we? I'm sure whatever the context is, it is deeply personal. Do you also have your popcorn ready?

Thank you. Yes, I'm going to refrain from airing out my dirty laundry. I made a bad decision, now I'm living with it, and more context doesn't actually change the intent behind my message: these tools are dangerous. Getting better, but still dangerous.

Re: AI overly affirms users asking for personal advice

#339

Post by epistasis »

zulban wrote: post #333
epistasis wrote: post #265

Earlier quoted context omitted.

I'm not so sure of that opinion on reproducibility. The last peer review I did was for a small journal that explicitly does not evaluate for high scientific significance, merely for correctness, which generally means straightforward acceptance. The other two reviews were positive, as was mine, except I said that the methods need to be described more and ideally the code placed somewhere. That was enough for a complet…

I'm not sure how one example contradicts documented huge overall trends, but okay.

I think publishers care about this a lot, but most researchers do not seem to care as much about reproducibility.

Re: AI overly affirms users asking for personal advice

#340

Post by tavavex »

user_7832 wrote: post #306
NortySpock wrote: post #216

Earlier quoted context omitted.

One mental model I have with LLMs is that they have been the subject of extreme evolutionary selection forces that are entirely the result of human preferences. Any LLM not sufficiently likable and helpful in the first two minutes was deleted or not further iterated on, or had so much retraining (sorry, "backpropagation") it's not the same as it started out. So it's going to say whatever it "thinks" you want it to sa…

Fully agree. I wonder in the long term how this will show up. Will every business/CEO do more of what he/they anyway want to do, but now supported by AI/LLMs? The possibilities in "dangerous" fields are a bit more frightening. A general is much more likely to ask ChatGPT "Do you think this war is a good idea/should I drop a bomb", rather than an actually helpful tool - where you might ask "What are 5 hidden points on…

> Will every business/CEO do more of what he/they anyway want to do, but now supported by AI/LLMs?

Arguably, it already worked that way. The best way to climb the ranks of a 'dictatorial' organization (a repressive government or an average large business) is to always say yes. Adopt what the people from up above want you to use, say and think. Don't question anything. Find silver linings in their most deranged ideas to show your loyalty. The rich and powerful that occupy the top ranks of these structures often hate being challenged, even if it's irrational for their well-being. Whenever you see a country or a company making a massive mistake, you can often trace it to a consequence of this. Humans hate being challenged and the rich can insulate themselves even further from the real world.

What's worrying me is the opposite - that this power is more available now. Instead of requiring a team of people and an asset cushion that lets you act irrationally, now you just need to have a phone in your pocket. People get addicted to LLMs because they can provide endless, varied validation for just about anything. Even if someone is aware of their own biases, it's not a given that they'll always counteract the validation.

Post reply on HN