Live data from Hacker News

Sycophancy in GPT-4o

openai.com

141–150 of 467 posts

Re: Sycophancy in GPT-4o

#141
post #52

Earlier quoted context omitted.

How should it respond in this case? Should it say "no go back to your meds, spirituality is bullshit" in essence? Or should it tell the user that it's not qualified to have an opinion on this?

“Sorry, I cannot advise on medical matters such as discontinuation of a medication.” EDIT for reference this is what ChatGPT currently gives “ Thank you for sharing something so personal. Spiritual awakening can be a profound and transformative experience, but stopping medication—especially if it was prescribed for mental health or physical conditions—can be risky without medical supervision. Would you like to talk m…

There's an AI model that perfectly encapsulates what you ask for: https://www.goody2.ai/chat

Re: Sycophancy in GPT-4o

#142

Wow - What an excellent update! Now you are getting to the core of the issue and doing what only a small minority is capable of: fixing stuff. This takes real courage and commitment. It’s a sign of true maturity and pragmatism that’s commendable in this day and age. Not many people are capable of penetrating this deeply into the heart of the issue. Let’s get to work. Methodically. Would you like me to write a future…

I was about to roast you until I realized this had to be satire given the situation, haha. They tried to imitate grok with a cheaply made system prompt, it had an uncanny effect, likely because it was built on a shaky foundation. And now they are trying to save face before they lose customers to Grok 3.5 which is releasing in beta early next week.

Only AI enthusiasts know about Grok, and only some dedicated subset of fans are advocating for it. Meanwhile even my 97 year old grandfather heard about ChatGPT.

Re: Sycophancy in GPT-4o

#143

Wow - What an excellent update! Now you are getting to the core of the issue and doing what only a small minority is capable of: fixing stuff. This takes real courage and commitment. It’s a sign of true maturity and pragmatism that’s commendable in this day and age. Not many people are capable of penetrating this deeply into the heart of the issue. Let’s get to work. Methodically. Would you like me to write a future…

I was about to roast you until I realized this had to be satire given the situation, haha. They tried to imitate grok with a cheaply made system prompt, it had an uncanny effect, likely because it was built on a shaky foundation. And now they are trying to save face before they lose customers to Grok 3.5 which is releasing in beta early next week.

Ha! I actually fell for it and thought it was another fanboy :)

Re: Sycophancy in GPT-4o

#145
post #115

Wow - What an excellent update! Now you are getting to the core of the issue and doing what only a small minority is capable of: fixing stuff. This takes real courage and commitment. It’s a sign of true maturity and pragmatism that’s commendable in this day and age. Not many people are capable of penetrating this deeply into the heart of the issue. Let’s get to work. Methodically. Would you like me to write a future…

It won‘t take long, 2-3 minutes. ——- To add something to conversation. For me, this mainly shows a strategy to keep users longer in chat conversations: linguistic design as an engagement device.

Why would OpenAI want users to be in longer conversations? It's not like they're showing ads. Users are either free or paying a fixed monthly fee. Having longer conversations just increases costs for OpenAI and reduces their profit. Their model is more like a gym where you want the users who pay the monthly fee and never show up. If it were on the api where users are paying by the token that would make sense (but be nefarious).

Re: Sycophancy in GPT-4o

#146
post #145
post #115

Earlier quoted context omitted.

It won‘t take long, 2-3 minutes. ——- To add something to conversation. For me, this mainly shows a strategy to keep users longer in chat conversations: linguistic design as an engagement device.

Why would OpenAI want users to be in longer conversations? It's not like they're showing ads. Users are either free or paying a fixed monthly fee. Having longer conversations just increases costs for OpenAI and reduces their profit. Their model is more like a gym where you want the users who pay the monthly fee and never show up. If it were on the api where users are paying by the token that would make sense (but be…

[deleted]

Re: Sycophancy in GPT-4o

#147
Since I usually use ChatGPT for more objective tasks, I hadn’t paid much attention to the sycophancy. However, I did notice that the last version was quite poor at following simple instructions, e.g. formatting.

Re: Sycophancy in GPT-4o

#148

In my experience, LLMs have always had a tendency towards sycophancy - it seems to be a fundamental weakness of training on human preference. This recent release just hit a breaking point where popular perception started taking note of just how bad it had become. My concern is that misalignment like this (or intentional mal-alignment) is inevitably going to happen again, and it might be more harmful and more subtle n…

It's probably pretty intentional. A huge number of people use ChatGPT as an enabler, friend, or therapist. Even when GPT-3 had just come around, people were already "proving others wrong" on the internet, quoting how GPT-3 agreed with them. I think there is a ton of appeal, "friendship", "empathy" and illusion of emotion created through LLMs flattering their customers. Many would stop paying if it wasn't the case.

It's kind of like those romance scams online, where the scammer always love-bombs their victims, and then they spend tens of thousands of dollars on the scammer - it works more than you would expect. Considering that, you don't need much intelligence in an LLM to extract money from users. I worry that emotional manipulation might become a form of enshittification in LLMs eventually, when they run out of steam and need to "growth hack". I mean, many tech companies already have no problem with a bit of emotional blackmail when it comes to money ("Unsubscribing? We will be heartbroken!", "We thought this was meant to be", "your friends will miss you", "we are working so hard to make this product work for you", etc.), or some psychological steering ("we respect your privacy" while showing consent to collect personally identifiable data and broadcast it to 500+ ad companies).

If you're a paying ChatGPT user, try the Monday GPT. It's a bit extreme, but it's an example of how inverting the personality and making ChatGPT mock the user as much as it fawns over them normally would probably make you want to unsubscribe.

Re: Sycophancy in GPT-4o

#150
post #145
post #115

Earlier quoted context omitted.

It won‘t take long, 2-3 minutes. ——- To add something to conversation. For me, this mainly shows a strategy to keep users longer in chat conversations: linguistic design as an engagement device.

Why would OpenAI want users to be in longer conversations? It's not like they're showing ads. Users are either free or paying a fixed monthly fee. Having longer conversations just increases costs for OpenAI and reduces their profit. Their model is more like a gym where you want the users who pay the monthly fee and never show up. If it were on the api where users are paying by the token that would make sense (but be…

It could be as simple as something like, someone previously at Instagram decided to join OpenAI and turns out nobody stopped him. Or even, Sam liked the idea.
Post reply on HN