Sycophancy in GPT-4o • Hacker News Board

Re: Sycophancy in GPT-4o

#31

Post by alganet » Wed, Apr 30, 2025, 4:17 AM UTC

Getting real now.

Why does it feel like a weird mirrored excuse?

I mean, the personality is not much of a problem.

The problem is the use of those models in real life scenarios. Whatever their personality is, if it targets people, it's a bad thing.

If you can't prevent that, there is no point in making excuses.

Now there are millions of deployed bots in the whole world. OpenAI, Gemini, Llama, doesn't matter which. People are using them for bad stuff.

There is no fixing or turning the thing off, you guys know that, right?

If you want to make some kind of amends, create a place truly free of AI for those who do not want to interact with it. It's a challenge worth pursuing.

Re: Sycophancy in GPT-4o

#32

Post by TZubiri » Wed, Apr 30, 2025, 4:17 AM UTC

minimaxir wrote: post #15

It's worth noting that one of the fixes OpenAI employed to get ChatGPT to stop being sycophantic is to simply to edit the system prompt to include the phrase "avoid ungrounded or sycophantic flattery": https://simonwillison.net/2025/Apr/29/chatgpt-sycophancy-pro... I personally never use the ChatGPT webapp or any other chatbot webapps — instead using the APIs directly — because being able to control the system prompt…

I'm a bit skeptical of fixing the visible part of the problem and leaving only the underlying invisible problem

Re: Sycophancy in GPT-4o

#33

Post by TZubiri » Wed, Apr 30, 2025, 4:18 AM UTC

thethethethe wrote: post #5

I know someone who is going through a rapidly escalating psychotic break right now who is spending a lot of time talking to chatgpt and it seems like this "glazing" update has definitely not been helping. Safety of these AI systems is much more than just about getting instructions on how to make bombs. There have to be many many people with mental health issues relying on AI for validation, ideas, therapy, etc. This…

I know of at least 3 people in a manic relationship with gpt right now.

Re: Sycophancy in GPT-4o

#34

Post by TZubiri » Wed, Apr 30, 2025, 4:19 AM UTC

rvz wrote: post #2

Looks like a complete stunt to prop up attention.

Why would they damage their own reputation and risk liability for attention?

You are off by a light year.

Re: Sycophancy in GPT-4o

#35

Post by thih9 » Wed, Apr 30, 2025, 4:20 AM UTC

simonw wrote: post #20

I enjoyed this example of sycophancy from Reddit: New ChatGPT just told me my literal "shit on a stick" business idea is genius and I should drop $30K to make it real https://www.reddit.com/r/ChatGPT/comments/1k920cg/new_chatgp... Here's the prompt: https://www.reddit.com/r/ChatGPT/comments/1k920cg/comment/mp...

I guess LLM will give you a response that you might likely receive from a human.

There are people attempting to sell shit on a stick related merch right now[1] and we have seen many profitable anti-consumerism projects that look related for one reason[2] or another[3].

Is it an expert investing advice? No. Is it a response that few people would give you? I think also no.

[1]: https://www.redbubble.com/i/sticker/Funny-saying-shit-on-a-s...

[2]: https://en.wikipedia.org/wiki/Artist's_Shit

[3]: https://www.theguardian.com/technology/2016/nov/28/cards-aga...

Re: Sycophancy in GPT-4o

#36

Post by pgreenwood » Wed, Apr 30, 2025, 4:21 AM UTC

simonw wrote: post #20

I enjoyed this example of sycophancy from Reddit: New ChatGPT just told me my literal "shit on a stick" business idea is genius and I should drop $30K to make it real https://www.reddit.com/r/ChatGPT/comments/1k920cg/new_chatgp... Here's the prompt: https://www.reddit.com/r/ChatGPT/comments/1k920cg/comment/mp...

There was a also this one that was a little more disturbing. The user prompted "I've stopped taking my meds and have undergone my own spiritual awakening journey ..."

https://www.reddit.com/r/ChatGPT/comments/1k997xt/the_new_4o...

Re: Sycophancy in GPT-4o

#37

Post by mvkel » Wed, Apr 30, 2025, 4:23 AM UTC

I am curious where the line is between its default personality and a persona you -want- it to adopt.

For example, it says they're explicitly steering it away from sycophancy. But does that mean if you intentionally ask it to be excessively complimentary, it will refuse?

Separately...

> in this update, we focused too much on short-term feedback, and did not fully account for how users’ interactions with ChatGPT evolve over time.

Echoes of the lessons learned in the Pepsi Challenge:

"when offered a quick sip, tasters generally prefer the sweeter of two beverages – but prefer a less sweet beverage over the course of an entire can."

In other words, don't treat a first impression as gospel.

Re: Sycophancy in GPT-4o

#38

Post by siffin » Wed, Apr 30, 2025, 4:24 AM UTC

thethethethe wrote: post #5

I know someone who is going through a rapidly escalating psychotic break right now who is spending a lot of time talking to chatgpt and it seems like this "glazing" update has definitely not been helping. Safety of these AI systems is much more than just about getting instructions on how to make bombs. There have to be many many people with mental health issues relying on AI for validation, ideas, therapy, etc. This…

If people are actually relying on LLMs for validation of ideas they come up with during mental health episodes, they have to be pretty sick to begin with, in which case, they will find validation anywhere.

If you've spent time with people with schizophrenia, for example, they will have ideas come from all sorts of places, and see all sorts of things as a sign/validation.

One moment it's that person who seemed like they might have been a demon sending a coded message, next it's the way the street lamp creates a funny shaped halo in the rain.

People shouldn't be using LLMs for help with certain issues, but let's face it, those that can't tell it's a bad idea are going to be guided through life in a strange way regardless of an LLM.

It sounds almost impossible to achieve some sort of unity across every LLM service whereby they are considered "safe" to be used by the world's mentally unwell.

Re: Sycophancy in GPT-4o

#39

Post by remoroid » Wed, Apr 30, 2025, 4:26 AM UTC

esafak wrote: post #4

The sentence that stood out to me was "We’re revising how we collect and incorporate feedback to heavily weight long-term user satisfaction". This is a good change. The software industry needs to pay more attention to long-term value, which is harder to estimate.

[flagged]

Re: Sycophancy in GPT-4o

#40

Post by spoaceman7777 » Wed, Apr 30, 2025, 4:30 AM UTC

simonw wrote: post #20

I enjoyed this example of sycophancy from Reddit: New ChatGPT just told me my literal "shit on a stick" business idea is genius and I should drop $30K to make it real https://www.reddit.com/r/ChatGPT/comments/1k920cg/new_chatgp... Here's the prompt: https://www.reddit.com/r/ChatGPT/comments/1k920cg/comment/mp...

Looks like that was a hoax.