Live data from Hacker News

Expanding on what we missed with sycophancy

openai.com

191–200 of 297 posts

Re: Expanding on what we missed with sycophancy

#191

Moments like these make me reevaluate the AI doomer view point. We aren't just toying with access to dangerous ideas (biological weapons, etc) we are toying with human psychology. If something as obvious as harmful sycophancy can slip out so easily, what subtle harms are being introduced. It's like lead in paint (and gasoline) except rewiring our very brains. We won't know the real problems for decades.

I still casually believe AI doomerism is valid, but it will rear its head in more depressing, incompetent ways:

- a broken AI market will cause another financial collapse via bubble

- broken AI products will get access to the wrong mission critical civil system, or at least a part of that call chain, and there will be some devastating loss. It won’t matter though, because it won’t affect the billionaire class.

- we’ll never achieve an actually singularity based on a superintelligence, but we’ll get AI weapons. Those AI weapons will be in the hands of sociopathic autocrats who view mankind in terms of what can be taken.

My general view is that we’re on the worst possible timeline and mankind has reverted back to our primate ancestry to make decisions: biggest strongest monkey wins. There is only law of jungle. Ook ook.

Re: Expanding on what we missed with sycophancy

#192

Moments like these make me reevaluate the AI doomer view point. We aren't just toying with access to dangerous ideas (biological weapons, etc) we are toying with human psychology. If something as obvious as harmful sycophancy can slip out so easily, what subtle harms are being introduced. It's like lead in paint (and gasoline) except rewiring our very brains. We won't know the real problems for decades.

There's some pretty foreseeable stuff just considering the existing attention capitalism business model of big tech we all know and loathe. Eventually OpenAI is going to have to make money, and blending ads into answers will be an obvious way. Next step will be maximizing eyeball time on those ads by any means necessary, including all the engagement baiting techniques Meta and other social media companies have alread…

They have already introduced ads btw.

“The "Enshittification" has arrived I asked ChatGPT about the impact of the current tarrifs on inventories over the next few months. It returned a long list of links to toiletries I might want to buy. I asked it why it did that. It replied: "As of April 28, 2025, OpenAl introduced new shopping features to ChatGPT, enhancing its capabilities to provide product recommendations complete with images, reviews, and direct purchase links. These features are available to all users, including those on Free, Plus, and Pro tiers, and even to users not logged in. The recommendations are generated organically, without paid advertisements or commission- based incentives, relying instead on structured metadata from third-party sources such as pricing, product descriptions, and reviews. This update aims to offer a more personalized and streamlined shopping experience directly within the ChatGPT interface, allowing users to explore products across various categories like fashion, beauty, electronics, and home goods. If you have any specific preferences or need tailored recommendations, feel free to let me know!"

Re: Expanding on what we missed with sycophancy

#193

Earlier quoted context omitted.

I see the same. I'm waiting on LLMs to get good enough that I can use them to help me learn foreign languages - e.g. talk to me about the news in language X. This way I can learn a language in an interesting and interactive way without burdening some poor human with my mistakes. I would build this myself but others will probably beat me too it.

LLMs are already good enough to tell you the news in language X and listen to your broken attempts at asking questions back. Or what's missing?

They do seem close to being able to, I'm mostly waiting on someone to provide it as a service - I don't have time for side projects ATM.

Re: Expanding on what we missed with sycophancy

#194
I, maybe embarrassingly, use chatgpt a lot for processing personal issues and journaling. It is legitimately helpful, and especially in helping me reword messages to people that have a lot of emotion underneath them, and making sure they are kind, communicative, and as objective as possible.

I am somewhat frustrated with openai's miss here, because during this time i was leaning heavily on chatgpt for a situation in my life that ultimately led to the end of a relationship. Chatgpt literally helped me write the letter that served as the brief and final conversation of that relationship. And while I stand by my decision and the reasons for it, I think it would have been very beneficial to get slightly more push back from my robot therapy sessions at the time. I did thankfully also have the foresight to specifically ask for it to find flaws, including by trying to pretend it was a breakup letter sent to me, so that maybe it would take the "other side".

Yes, I know, therapists and friends are a better option and chatgpt is not a substitute for real humans and human feedback. however this is something i spent weeks journaling and processing on and i wasnt about to ask anyone to give me that much time for a single topic like this. i did also ask friends for feedback, too. chatgpt has factually really helped me in several relationship situations in my life, i just want to know that the feedback im getting is inline with what i expect having worked with it so much

Re: Expanding on what we missed with sycophancy

#195

Earlier quoted context omitted.

>If it activates the same neural pathways, and has the same results, then I think the mind doesn't care Boiling it down to neural signals is a risky approach, imo. There are innumerable differences between these interactions. This isn't me saying interactions are inherently dangerous if artificial empathy is baked in, but equating them to real empathy is. Understanding those differences is critical, especially in a w…

Can you define "real empathy"?

There's a book that I encourage everyone to read called Motivational Interviewing. I've read the 3rd edition and I'm currently working my way through the 4th edition to see what's changed, because it's a textbook that they basically rewrite completely with each new edition.

Motivational Interviewing is an evidence-based clinical technique for helping people move through ambivalence during the contemplation, preparation, and action stages of change under the Transtheoretical Model.

In Chapter 2 of the 3rd Edition, they define Acceptance as one of the ingredients for change, part of the "affect" of Motivational Interviewing. Ironically, people do not tend to change when they perceive themselves as unacceptable as they are. It is when they feel accepted as they are that they are able to look at themselves without feeling defensive and see ways in which they can change and grow.

Nearly all that they describe in chapter 2 is affective—it is neither sufficient nor even necessary in the clinical context that the clinician feel a deep acceptance for the client within themselves, but the client should feel deeply accepted so that they are given an environment in which they can grow. The four components of the affect of acceptance are autonomy support, absolute worth (what Carl Rogers termed "Unconditional Positive Regard"), accurate empathy, and affirmation of strengths and efforts.

Chapters 5 and 6 of the third edition define the skills of providing the affect of acceptance defined in Chapter 2—again, not as a feeling, but as a skill. It is something that can be taught, practiced, and learned. It is a common misconception to believe that unusually accepting people become therapists, but what is actually the case is that practicing the skill of accurate empathy trains the practitioner to be unusually accepting.

The chief skill of accurate empathy is that of "reflective listening", which essentially consists of interpreting what the other person has said and saying your interpretation back to them as a statement. For an unskilled listener, this might be a literal rewording of what was said, but more skilled listeners can, when appropriate, offer reflections that read between the lines. Very skilled listeners (as measured by scales like the Therapist Empathy Scale) will occasionally offer reflections that the person being listened to did not think, but will recognize within themselves once they have heard it.

In that sense, in the way that we measure empathy in settings where it is clinically relevant, I've found that AIs are very capable with some prompting of displaying the affect of accurate empathy.

Re: Expanding on what we missed with sycophancy

#196
post #71
post #13

Earlier quoted context omitted.

Are you sure that was real? I thought it was an made up example of the problems with the update

There are several threads on Reddit. For example https://www.reddit.com/r/ChatGPT/comments/1kalae8/chatgpt_in... Perhaps everyone there is LARPing - but if you start typing stereotypical psychosis talk into ChatGPT, it won't be long before it starts agreeing with your divinity.

reddit is overwhelmingly fake content, like a massive percentage of it. a post on reddit these days is not actually evidence of anything real, at all

Re: Expanding on what we missed with sycophancy

#197

Earlier quoted context omitted.

Put simply, GPT has no information about its internals. There is no method for introspection like you might infer from human reasoning abilities. Expecting anything but an hallucination in this instance is wishful thinking. And in any case, the risk of hallucination more generally means you should really vet information further than an LLM before spreading that information about.

True, the LLM has no information but OpenAI has provided it with enough information to explain it's memory system in regards to Project folders. I tested this out. If you want a chat without chat memory start a blank project and chat in there. I also discovered experientially that chat history memory is not editable. These aren't hallucinations.

> I had a discussion with GPT 4o about the memory system.

This sentence is really all i'm criticizing. Can you hypothesize how the memory system works and then probe the system to gain better or worse confidence in your hypothesis? Yes. But that's not really what that first sentence implied. It implied that you straight up asked ChatGPT and took it on faith even though you can't even get a correct answer on the training cutoff date from ChatGPT (so they clearly aren't stuffing as much information into the system prompt as you might think, or they are but there's diminishing returns on the effectiveness)

Re: Expanding on what we missed with sycophancy

#198
post #3

OpenAI mentions the new memory features as a partial cause. My theory as a imperative/functional programmer is that those features added global state to prompts that didn't have it before leading to unpredictability and instabilty. Prompts went from stateless to stateful. As GPT 4o put it: 1. State introduces non-determinism across sessions 2. Memory + sycophancy is a feedback loop 3. Memory acts as a shadow prompt m…

It is. If you start a fresh chat, turn on advanced voice, and just make any random sound like snapping your fingers it will just randomly pick up as if you’re continuing some other chat with no context (on the user side). I honestly really dislike that it considers all my previous interactions because I typically used new chats as a way to get it out of context ruts.

Settings -> Personalization -> Memory -> Disable

https://help.openai.com/en/articles/8983136-what-is-memory

Re: Expanding on what we missed with sycophancy

#199

Earlier quoted context omitted.

It is. If you start a fresh chat, turn on advanced voice, and just make any random sound like snapping your fingers it will just randomly pick up as if you’re continuing some other chat with no context (on the user side). I honestly really dislike that it considers all my previous interactions because I typically used new chats as a way to get it out of context ruts.

I don't like the change either. At the least it should be an option you can configure. But, can you use a "temporary" chat to ignore your other chats as a workaround?

Settings -> Personalization -> Memory -> Disable

Re: Expanding on what we missed with sycophancy

#200

Earlier quoted context omitted.

True, the LLM has no information but OpenAI has provided it with enough information to explain it's memory system in regards to Project folders. I tested this out. If you want a chat without chat memory start a blank project and chat in there. I also discovered experientially that chat history memory is not editable. These aren't hallucinations.

> I had a discussion with GPT 4o about the memory system. This sentence is really all i'm criticizing. Can you hypothesize how the memory system works and then probe the system to gain better or worse confidence in your hypothesis? Yes. But that's not really what that first sentence implied. It implied that you straight up asked ChatGPT and took it on faith even though you can't even get a correct answer on the train…

We're in different modes. I'm still feeling the glow of the thing coming alive and riffing on how perhaps its the memory change and you're interested in a different conversation.

Part of my process is to imagine I'm having a conversation like Hanks and Wilson, or a coderand a rubber duck, but you want to tell me Wilson is just a volleyball and the duck can't be trusted.

Post reply on HN