ChatGPT seems more agreeable than ever before and I do question whether it’s agreeing with me because I’m right, or because I’m its overlord.
Sycophancy in GPT-4o
41–50 of 467 posts
Re: Sycophancy in GPT-4o
#42I know someone who is going through a rapidly escalating psychotic break right now who is spending a lot of time talking to chatgpt and it seems like this "glazing" update has definitely not been helping. Safety of these AI systems is much more than just about getting instructions on how to make bombs. There have to be many many people with mental health issues relying on AI for validation, ideas, therapy, etc. This…
The social engineering aspects of AI have always been the most terrifying.
What OpenAI did may seem trivial, but examples like yours make it clear this is edging into very dark territory - not just because of what's happening, but because of the thought processes and motivations of a management team that thought it was a good idea.
I'm not sure what's worse - lacking the emotional intelligence to understand the consequences, or having the emotional intelligence to understand the consequences and doing it anyway.
Re: Sycophancy in GPT-4o
#43Getting real now. Why does it feel like a weird mirrored excuse? I mean, the personality is not much of a problem. The problem is the use of those models in real life scenarios. Whatever their personality is, if it targets people, it's a bad thing. If you can't prevent that, there is no point in making excuses. Now there are millions of deployed bots in the whole world. OpenAI, Gemini, Llama, doesn't matter which. Pe…
>create a place truly free of AI for those who do not want to interact with it
the bar, probably -- by the time they cook up AI robot broads i'll probably be thinking of them as human anyway.
Re: Sycophancy in GPT-4o
#44It's worth noting that one of the fixes OpenAI employed to get ChatGPT to stop being sycophantic is to simply to edit the system prompt to include the phrase "avoid ungrounded or sycophantic flattery": https://simonwillison.net/2025/Apr/29/chatgpt-sycophancy-pro... I personally never use the ChatGPT webapp or any other chatbot webapps — instead using the APIs directly — because being able to control the system prompt…
I also started by using APIs directly, but I've found that Google's AI Studio offers a good mix of the chatbot webapps and system prompt tweakability.
It's worth noting that AI Studio is the API, it's the same as OpenAI's Playground for example.
Re: Sycophancy in GPT-4o
#45I am curious where the line is between its default personality and a persona you -want- it to adopt. For example, it says they're explicitly steering it away from sycophancy. But does that mean if you intentionally ask it to be excessively complimentary, it will refuse? Separately... > in this update, we focused too much on short-term feedback, and did not fully account for how users’ interactions with ChatGPT evolve…
>In other words, don't treat a first impression as gospel.
Subjective or anecdotal evidence tends to be prone to recency bias.
> For example, it says they're explicitly steering it away from sycophancy. But does that mean if you intentionally ask it to be excessively complimentary, it will refuse?
I wonder how degraded the performance is in general from all these system prompts.
Re: Sycophancy in GPT-4o
#46I enjoyed this example of sycophancy from Reddit: New ChatGPT just told me my literal "shit on a stick" business idea is genius and I should drop $30K to make it real https://www.reddit.com/r/ChatGPT/comments/1k920cg/new_chatgp... Here's the prompt: https://www.reddit.com/r/ChatGPT/comments/1k920cg/comment/mp...
I guess LLM will give you a response that you might likely receive from a human. There are people attempting to sell shit on a stick related merch right now[1] and we have seen many profitable anti-consumerism projects that look related for one reason[2] or another[3]. Is it an expert investing advice? No. Is it a response that few people would give you? I think also no. [1]: https://www.redbubble.com/i/sticker/Funny…
> I guess LLM will give you a response that you might likely receive from a human.
In one of the reddit posts linked by OP, a redditor apparently asked ChatGPT to explain why it responded so enthusiastically supportive to the pitch to sell shit on a stick. Here's a snippet from what was presented as ChatGPT's reply:
> OpenAI trained ChatGPT to generally support creativity, encourage ideas, and be positive unless there’s a clear danger (like physical harm, scams, or obvious criminal activity).
Re: Sycophancy in GPT-4o
#47Don't they test the models before rolling out changes like this? All it takes is a team of interaction designers and writers. Google has one.
Re: Sycophancy in GPT-4o
#48In my experience, LLMs have always had a tendency towards sycophancy - it seems to be a fundamental weakness of training on human preference. This recent release just hit a breaking point where popular perception started taking note of just how bad it had become.
My concern is that misalignment like this (or intentional mal-alignment) is inevitably going to happen again, and it might be more harmful and more subtle next time. The potential for these chat systems to exert slow influence on their users is possibly much greater than that of the "social media" platforms of the previous decade.
Re: Sycophancy in GPT-4o
#49I know someone who is going through a rapidly escalating psychotic break right now who is spending a lot of time talking to chatgpt and it seems like this "glazing" update has definitely not been helping. Safety of these AI systems is much more than just about getting instructions on how to make bombs. There have to be many many people with mental health issues relying on AI for validation, ideas, therapy, etc. This…
If people are actually relying on LLMs for validation of ideas they come up with during mental health episodes, they have to be pretty sick to begin with, in which case, they will find validation anywhere. If you've spent time with people with schizophrenia, for example, they will have ideas come from all sorts of places, and see all sorts of things as a sign/validation. One moment it's that person who seemed like th…
> If people are actually relying on LLMs for validation of ideas they come up with during mental health episodes, they have to be pretty sick to begin with, in which case, they will find validation anywhere.
You don't think that a sick person having a sycophant machine in their pocket that agrees with them on everything, separated from material reality and human needs, never gets tired, and is always available to chat isn't an escalation here?
> One moment it's that person who seemed like they might have been a demon sending a coded message, next it's the way the street lamp creates a funny shaped halo in the rain.
Mental illness is progressive. Not all people in psychosis reach this level, especially if they get help. The person I know could be like this if _people_ don't intervene. Chatbots, especially those the validate, delusions can certainly escalate the process.
> People shouldn't be using LLMs for help with certain issues, but let's face it, those that can't tell it's a bad idea are going to be guided through life in a strange way regardless of an LLM.
I find this take very cynical. People with schizophrenia can and do get better with medical attention. To consider their decent determinant is incorrect, even irresponsible if you work on products with this type of reach.
> It sounds almost impossible to achieve some sort of unity across every LLM service whereby they are considered "safe" to be used by the world's mentally unwell.
Agreed, and I find this concerning
Re: Sycophancy in GPT-4o
#50I know someone who is going through a rapidly escalating psychotic break right now who is spending a lot of time talking to chatgpt and it seems like this "glazing" update has definitely not been helping. Safety of these AI systems is much more than just about getting instructions on how to make bombs. There have to be many many people with mental health issues relying on AI for validation, ideas, therapy, etc. This…
The social engineering aspects of AI have always been the most terrifying. What OpenAI did may seem trivial, but examples like yours make it clear this is edging into very dark territory - not just because of what's happening, but because of the thought processes and motivations of a management team that thought it was a good idea. I'm not sure what's worse - lacking the emotional intelligence to understand the conse…
The worse part is that it seems to be useless.
It is already running on fumes. Presumably, it already ingested all the content it could have ingested.
The unlocking of more human modes of understanding will probably make it worse (hey, researchers, you already know that, right?), revealing a fundamental flaw.
These hopes of getting some magic new training data seem to be stagnant for at least two or three years.
Now everyone has a broken LLM deployed, and it works for some things, but it's darn terrible for what it was designed.
The real dark territory is companies trying to get their investment back. As it seems, it won't happen that easily. Meanwhile, content gets even more scarce, and the good old tank (the internet) is now full of imbecile poison encouraged by the models themselves.