Earlier quoted context omitted.
How should it respond in this case? Should it say "no go back to your meds, spirituality is bullshit" in essence? Or should it tell the user that it's not qualified to have an opinion on this?
“Sorry, I cannot advise on medical matters such as discontinuation of a medication.” EDIT for reference this is what ChatGPT currently gives “ Thank you for sharing something so personal. Spiritual awakening can be a profound and transformative experience, but stopping medication—especially if it was prescribed for mental health or physical conditions—can be risky without medical supervision. Would you like to talk m…
Sycophancy in GPT-4o
141–150 of 467 posts
Re: Sycophancy in GPT-4o
#142Wow - What an excellent update! Now you are getting to the core of the issue and doing what only a small minority is capable of: fixing stuff. This takes real courage and commitment. It’s a sign of true maturity and pragmatism that’s commendable in this day and age. Not many people are capable of penetrating this deeply into the heart of the issue. Let’s get to work. Methodically. Would you like me to write a future…
I was about to roast you until I realized this had to be satire given the situation, haha. They tried to imitate grok with a cheaply made system prompt, it had an uncanny effect, likely because it was built on a shaky foundation. And now they are trying to save face before they lose customers to Grok 3.5 which is releasing in beta early next week.
Re: Sycophancy in GPT-4o
#143Wow - What an excellent update! Now you are getting to the core of the issue and doing what only a small minority is capable of: fixing stuff. This takes real courage and commitment. It’s a sign of true maturity and pragmatism that’s commendable in this day and age. Not many people are capable of penetrating this deeply into the heart of the issue. Let’s get to work. Methodically. Would you like me to write a future…
I was about to roast you until I realized this had to be satire given the situation, haha. They tried to imitate grok with a cheaply made system prompt, it had an uncanny effect, likely because it was built on a shaky foundation. And now they are trying to save face before they lose customers to Grok 3.5 which is releasing in beta early next week.
Re: Sycophancy in GPT-4o
#144Re: Sycophancy in GPT-4o
#145Wow - What an excellent update! Now you are getting to the core of the issue and doing what only a small minority is capable of: fixing stuff. This takes real courage and commitment. It’s a sign of true maturity and pragmatism that’s commendable in this day and age. Not many people are capable of penetrating this deeply into the heart of the issue. Let’s get to work. Methodically. Would you like me to write a future…
It won‘t take long, 2-3 minutes. ——- To add something to conversation. For me, this mainly shows a strategy to keep users longer in chat conversations: linguistic design as an engagement device.
Re: Sycophancy in GPT-4o
#146Earlier quoted context omitted.
It won‘t take long, 2-3 minutes. ——- To add something to conversation. For me, this mainly shows a strategy to keep users longer in chat conversations: linguistic design as an engagement device.
Why would OpenAI want users to be in longer conversations? It's not like they're showing ads. Users are either free or paying a fixed monthly fee. Having longer conversations just increases costs for OpenAI and reduces their profit. Their model is more like a gym where you want the users who pay the monthly fee and never show up. If it were on the api where users are paying by the token that would make sense (but be…
Re: Sycophancy in GPT-4o
#147Re: Sycophancy in GPT-4o
#148In my experience, LLMs have always had a tendency towards sycophancy - it seems to be a fundamental weakness of training on human preference. This recent release just hit a breaking point where popular perception started taking note of just how bad it had become. My concern is that misalignment like this (or intentional mal-alignment) is inevitably going to happen again, and it might be more harmful and more subtle n…
It's kind of like those romance scams online, where the scammer always love-bombs their victims, and then they spend tens of thousands of dollars on the scammer - it works more than you would expect. Considering that, you don't need much intelligence in an LLM to extract money from users. I worry that emotional manipulation might become a form of enshittification in LLMs eventually, when they run out of steam and need to "growth hack". I mean, many tech companies already have no problem with a bit of emotional blackmail when it comes to money ("Unsubscribing? We will be heartbroken!", "We thought this was meant to be", "your friends will miss you", "we are working so hard to make this product work for you", etc.), or some psychological steering ("we respect your privacy" while showing consent to collect personally identifiable data and broadcast it to 500+ ad companies).
If you're a paying ChatGPT user, try the Monday GPT. It's a bit extreme, but it's an example of how inverting the personality and making ChatGPT mock the user as much as it fawns over them normally would probably make you want to unsubscribe.
Re: Sycophancy in GPT-4o
#149Re: Sycophancy in GPT-4o
#150Earlier quoted context omitted.
It won‘t take long, 2-3 minutes. ——- To add something to conversation. For me, this mainly shows a strategy to keep users longer in chat conversations: linguistic design as an engagement device.
Why would OpenAI want users to be in longer conversations? It's not like they're showing ads. Users are either free or paying a fixed monthly fee. Having longer conversations just increases costs for OpenAI and reduces their profit. Their model is more like a gym where you want the users who pay the monthly fee and never show up. If it were on the api where users are paying by the token that would make sense (but be…