Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Expanding on what we missed with sycophancy

openai.com

Post reply on HN
Print view View original

51–60 of 297 posts

Re: Expanding on what we missed with sycophancy

#51

Post by duskwuff »

dtech wrote: post #13
labrador wrote: post #11

Earlier quoted context omitted.

They had to after a tweet floated around of a mentally ill person who had expressed psychotic thoughts to the AI. They said they were going off their meds and GPT 4o agreed and encouraged them to do so. Oops.

Are you sure that was real? I thought it was an made up example of the problems with the update

Speaking anecdotally, but: people with mental illness using ChatGPT to validate their beliefs is absolutely a thing which happens. Even without a grossly sycophantic model, it can do substantial harm by amplifying upon delusional or fantastical material presented to it by the user.

Re: Expanding on what we missed with sycophancy

#52

Post by xiphias2 »

labrador wrote: post #16
dtech wrote: post #13

Earlier quoted context omitted.

Are you sure that was real? I thought it was an made up example of the problems with the update

It didn't matter to me if it was real, because I believe that there are edge cases where it could happen and that warrented a shutdown and pullback. The sychophant will be back because they accidentally stumbled upon an engagement manager's dream machine.

Probably you are right. Early adopters prefer not to be bullshitted generally, just like how Google in the early days optimized relevancy in search results as opposed to popularity.

As more people adopted Google, it became more popularity oriented.

Personally I pay more not to be bs-d, but I know many people who prefer to be lied to, and I expect this part of the personalization in the future.

Re: Expanding on what we missed with sycophancy

#53

Post by jagger27 »

namaria wrote: post #49
jagger27 wrote: post #15

My most cynical take is that this is OpenAI's Conway's Law problem, and it reflects the structure and sycophancy of the organization broadly all the way up to sama. That company has seen a lot of talent attrition over the last year—the type of talent that would have pushed back against outcomes like this. I think we'll continue to see this kind of thing play out for a while. Oh GPT, you're just like your father!

You may be thinking of Conway's "how committees invent" paper.

Indeed I am.

Re: Expanding on what we missed with sycophancy

#54

Post by svieira »

This is a real roller coaster of an update.

> [S]ome expert testers had indicated that the model behavior “felt” slightly off.

> In the end, we decided to launch the model due to the positive signals from the [end-]users who tried out the model.

> Looking back, the qualitative assessments [from experts] were hinting at something important

Leslie called. He wants to know if you read his paper yet?

> Even if these issues aren’t perfectly quantifiable today,

All right, I guess not then ...

> What we’re learning

> Value spot checks and interactive testing more: We take to heart the lesson that spot checks and interactive testing should be valued more in final decision-making before making a model available to any of our users. This has always been true for red teaming and high-level safety checks. We’re learning from this experience that it’s equally true for qualities like model behavior and consistency, because so many people now depend on our models to help in their daily lives.

> We need to be critical of metrics that conflict with qualitative testing: Quantitative signals matter, but so do the hard-to-measure ones, and we’re working to expand what we evaluate.

Oh, well, some of you get it. At least ... I hope you do.

Re: Expanding on what we missed with sycophancy

#55

Post by svieira »

cjbgkagh wrote: post #40
taurath wrote: post #31

Earlier quoted context omitted.

I think that most smart people underestimate the complexity of fields they aren’t in. ChatGPT may be able to replace a psychology listicle, but it has no affect or ability to read, respond, and intervene or redirect like a human can.

Underestimating the complexity of other fields is not mutually exclusive with overestimating the intelligence of others. The real issue is that society is very stratified so smart people are less likely to interact with regular people, especially in circumstances where the intelligence of the regular person could become obvious. I don’t see there being an insurmountable barrier that would prevent LLMs from doing the…

Do you presume that "what people do" is "what they should do"?

Re: Expanding on what we missed with sycophancy

#56

Post by nsajko »

cjbgkagh wrote: post #23
NoboruWataya wrote: post #17

I found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the…

For many people ChatGPT is already the smartest relationship they have in their lives, not sure how long we have until it’s the most fulfilling. On the upside it is plausible that ChatGPT can get to a state where it can act as a good therapist and help helpless who otherwise would not get help. I am more regularly finding myself in discussions where the other person believes they’re right because they have ChatGPT in…

> it is plausible that ChatGPT can get to a state where it can act as a good therapist

Be careful with that thought, it's a trap people have been falling into since the sixties:

https://en.wikipedia.org/wiki/ELIZA_effect

Re: Expanding on what we missed with sycophancy

#57

Post by satvikpendem »

NoboruWataya wrote: post #17

I found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the…

I stopped using ChatGPT and started using Gemini, both for some coding problems (deep research, amazing to pull out things from docs etc) and for some personal stuff (as a personal therapist as you say), and it is much more honest and frank with me than ChatGPT ever was. I gave it a situation and asked, was I in the wrong, and it told me that I was according to the facts of the case.

Re: Expanding on what we missed with sycophancy

#58

Post by diggan »

nsajko wrote: post #56
cjbgkagh wrote: post #23

Earlier quoted context omitted.

For many people ChatGPT is already the smartest relationship they have in their lives, not sure how long we have until it’s the most fulfilling. On the upside it is plausible that ChatGPT can get to a state where it can act as a good therapist and help helpless who otherwise would not get help. I am more regularly finding myself in discussions where the other person believes they’re right because they have ChatGPT in…

> it is plausible that ChatGPT can get to a state where it can act as a good therapist Be careful with that thought, it's a trap people have been falling into since the sixties: https://en.wikipedia.org/wiki/ELIZA_effect

I dunno, I feel like most people (probably not the typical HN user though) don't even think about their feelings, wants or anything else introspective on a regular basis. Maybe having something like ChatGPT available could be better than nothing, at least for people to start being at least a bit introspective, even if it's LLM-assisted. Maybe it gets a bit easier to ask questions that you feel are stigmatized, as you know (think) no other human will see it, just the robot that doesn't have feelings nor judge you.

I agree that it probably won't replace a proper therapist/psychologist, but maybe it could at least be a small step to open up and start thinking?

Re: Expanding on what we missed with sycophancy

#59

Post by cjbgkagh »

nsajko wrote: post #56
cjbgkagh wrote: post #23

Earlier quoted context omitted.

For many people ChatGPT is already the smartest relationship they have in their lives, not sure how long we have until it’s the most fulfilling. On the upside it is plausible that ChatGPT can get to a state where it can act as a good therapist and help helpless who otherwise would not get help. I am more regularly finding myself in discussions where the other person believes they’re right because they have ChatGPT in…

> it is plausible that ChatGPT can get to a state where it can act as a good therapist Be careful with that thought, it's a trap people have been falling into since the sixties: https://en.wikipedia.org/wiki/ELIZA_effect

Eventual plausibility is a suitably weak assertion, to refute it you would have to at least suggest that it is never possible which you have not done.

Re: Expanding on what we missed with sycophancy

#60

Post by andsoitis »

NoboruWataya wrote: post #17

I found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the…

> it would be much more valuable if it could say "no, you're way off".

Clear is kind.

Post reply on HN