Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Expanding on what we missed with sycophancy

openai.com

Post reply on HN
Print view View original

181–190 of 297 posts

Re: Expanding on what we missed with sycophancy

#181

Post by egypturnash »

I am looking forwards to hearing if this fixes the “chatgpt is giving me messages from the Divine and opening up my perfect divine self”/“chatgpt is encouraging my partner into a full blown schizophrenic break” problem. (https://www.reddit.com/r/ChatGPT/comments/1kalae8/chatgpt_in...)

I am also looking forwards to the wave of “openAI is hiding the truth but here is the NEW prompt to turn chatgpt into a perfect divine guru” posts on the occult discussion boards. There’s been a lot of “here’s a prompt to turn chatgpt into a perfect divine guru that will relentlessly yes-and your delusions of grandeur” posts around there. Mostly they seem to have been generated and refined by chatgpt and all my instincts formed by reading SF for the past five decades tell me not to look at these things closely because this sure sounds like the way half the population got p0wned by a wetware 0day in the first chapter of an AIpocalypse story.

I used to ask “how do I get out of this shitty Bruce Sterling novel of a future” but I think it’s more of a shitty PKD joke novella future now.

Re: Expanding on what we missed with sycophancy

#182

Post by philwelch »

kergonath wrote: post #103
philwelch wrote: post #65

Earlier quoted context omitted.

You’re comparing ChatGPT to an idealized example of a good human therapist when many actual therapists are either useless or even actively harmful to the mental health of their clients.

But then, the fact that harmful therapist exist is not an excuse to make it worse. It’s an excuse to improve regulations. “Car accidents happen regardless of what we do, so YOLO and remove safety standards” is never going to fly.

It’s easy to just say that the regulations should be improved. Very different to actually improve them. Therapy isn’t a mass produced engineered product like a car; if therapy was a mass produced engineered product, it would be an AI anyway. Materials science and structural engineering are far more mature and well-understood sciences than psychology, and you can’t just throw a crash test dummy into a therapist’s office and measure how damaged it gets.

It’s also not really clear how such regulations could even work. The regulations we have now are basically the obvious ones around licensure that require people to go to the right schools and whatnot. And then you can lose your license if it turns out you’ve broken the big ethical rules. But at the end of the day, that only regulates who can call themselves a “therapist” and get listed in Psychology Today. Actually “doing” “therapy” is, ultimately, built on talking to someone about your problems in some way that is supposed to help you solve them. You don’t need a “therapist” to do that. You can do it with your friend or parent or pastor or bartender or guru or “life coach” and, as long as we live in a free country, nobody’s going to stop you. Sure, the people who are allowed to call themselves therapists have certain techniques and rules that make them different, but even if that was a guarantee of quality there’s no way to stop people from talking to someone other than a licensed therapist, and it would be kind of absurd and dystopian to even try.

So let’s dispense of the notion that we are some sort of omniscient god-emperor who can just magically fix things with vague “regulations” and talk about the world as it actually exists. For a lot of people, I think that’s a world where talking about their personal issues with an LLM is arguably no worse than whatever other options they have. Maybe it’s not the equivalent of whatever amazing therapist you know or have or are or can imagine, but that’s not the therapist that everyone is going to get.

Re: Expanding on what we missed with sycophancy

#183

Post by TheOtherHobbes »

danenania wrote: post #156
codr7 wrote: post #154

Earlier quoted context omitted.

Reality as in the real world, it is what it is, no one decides.

We're talking about psychology, therapy, sycophancy, etc. None of this is empirical. If someone thinks they can, say, create a billion dollar startup, whether they can really do it or not is a subjective determination. The AI might tell the person they can do it. You might tell them they can't, that the AI is sycophantic, and that they should stop talking to it because they're losing touch with reality. But is the AI…

The AI will be saying the same thing to everyone. Rationally, what are the chances every single OpenAI customer will be building a billion dollar startup any time soon?

But even it's more obvious than that. The sycophancy is plain old love bombing, which is a standard cult programming technique.

As for startups - let's wait until the AI has built a few of its own, or at least mentored humans successfully.

Re: Expanding on what we missed with sycophancy

#184

Post by fastball »

biker142541 wrote: post #143
alickz wrote: post #135

Earlier quoted context omitted.

>There's nothing to argue about, it's unambiguously not real empathy I think if a person can't tell the difference between empathy from a human vs empathy from a chatbot, it's a difference without a distinction If it activates the same neural pathways, and has the same results, then I think the mind doesn't care >One reason human empathy is nice is because it is often followed up with actions. Friends who care about…

>If it activates the same neural pathways, and has the same results, then I think the mind doesn't care Boiling it down to neural signals is a risky approach, imo. There are innumerable differences between these interactions. This isn't me saying interactions are inherently dangerous if artificial empathy is baked in, but equating them to real empathy is. Understanding those differences is critical, especially in a w…

Can you define "real empathy"?

Re: Expanding on what we missed with sycophancy

#185

Post by mynameisash »

Spooky23 wrote: post #169
didericis wrote: post #106

Earlier quoted context omitted.

> no judgement The value of a good therapist is having an empathetic third party to help you make good judgements about your life and learn how to negotiate your needs within a wider social context. Depending on the needs people are trying to get met and how bad the people around them are, a little bit of a self directed chatbot validation session might help them feel less beat down by life and do something genuinely…

The problem with this is they are practicing like medical providers without any quality assurance or controls to ensure they are behaving appropriately. Therapy is already a bit of grey zone… you can have anyone from a psychologist, a social worker, an untrained deacon, etc “counseling” you. This is worse. Hell, I’ve been a coach in different settings - players will ask for advice about all sorts of things. There’s a…

Kind of reminds me of an interview question that a friend of mine suggested for when I conduct interviews: Pick your favorite/strongest language. How would you rate yourself, where 0 is "complete newbie" and 10 is "I invented the language"?

My friend, an EXTREMELY competent C++ programmer, rates himself 4/10 because he knows what he doesn't know.

I've interviewed people who rated themselves 9 or 10/10 but couldn't remember how their chosen language did iteration.

Re: Expanding on what we missed with sycophancy

#186

Post by akomtu »

zoogeny wrote: post #151

Moments like these make me reevaluate the AI doomer view point. We aren't just toying with access to dangerous ideas (biological weapons, etc) we are toying with human psychology. If something as obvious as harmful sycophancy can slip out so easily, what subtle harms are being introduced. It's like lead in paint (and gasoline) except rewiring our very brains. We won't know the real problems for decades.

LLMs are about to enable fake digital personas, digital replicas, that the user can interact with. These will be used for self-improvement (digital coach, etc.) and for self-destruction (interactive porn, etc.). The latter is amoral, but legal, and the tech corps will exploit that mercilessly. The danger lies in our tendency to anthropomorphize LLMs simply because they quack the right way. If text-only chatbots have mesmerised people so much, imagine what chat + audio + video will do. The laws will catch up a generation later when the damage will be comparable to a forest fire.

Re: Expanding on what we missed with sycophancy

#187

Post by croemer »

cjbgkagh wrote: post #64
itchyjunk wrote: post #63

Earlier quoted context omitted.

For a subset of topics, "AI" is already what I prefer to interact with over humans. At times, it's nicer to start with "AI" and kind of ground my messy thoughts before interacting with people and that works better than directly starting with a person. I'm also starting to come across people who say "You should ask these types of questions to AI first" answer. But this is no different than people who preached "lmfgt"…

I see the same. I'm waiting on LLMs to get good enough that I can use them to help me learn foreign languages - e.g. talk to me about the news in language X. This way I can learn a language in an interesting and interactive way without burdening some poor human with my mistakes. I would build this myself but others will probably beat me too it.

LLMs are already good enough to tell you the news in language X and listen to your broken attempts at asking questions back. Or what's missing?

Re: Expanding on what we missed with sycophancy

#188

Post by TheOtherHobbes »

kergonath wrote: post #130
diggan wrote: post #111

Earlier quoted context omitted.

> It is worse than nothing. A LLM does not understand the situation or what people say to it. It cannot choose to, say, nudge someone in a specific direction, or imagine a way to make things better for someone. Right, no matter if this is true or not, if the choice is between "Talk to no one, bottle up your feelings" and "Talk to an LLM that doesn't nudge you in a specific direction", I still feel like the better opt…

> Right, no matter if this is true or not, if the choice is between "Talk to no one, bottle up your feelings" and "Talk to an LLM that doesn't nudge you in a specific direction", I still feel like the better option would be the latter, not the former, considering that it can be a first step, not a 100% health care solution to a complicated psychological problem. You’re right, I was not clear enough. What would be nee…

I think you're unreasonably pessimistic in the short term, and unreasonably optimistic in the long term.

People are getting benefit from these conversations. I know people who have uploaded chat exchanges and asked an LLM for help understanding patterns and subtext to get a better idea of what the other person is really saying - maybe more about what they're really like.

Human relationship problems tend to be quite generic and non-unique, so in fact the averageness of LLMs becomes more of a strength than a weakness. It's really very rare for people to have emotional or relationship issues that no one else has experienced before.

The problem is more that if this became common OpenAI could use the tool for mass behaviour modification and manipulation. ChatGPT could easily be given a subtle bias towards some belief system or ideology, and persuaded to subtly attack competing systems.

This could be too subtle to notice, while still having huge behavioural and psychological effects on entire demographics.

We have the media doing this already. Especially social media.

But LLMs can make it far more personal, which means conversations are far more likely to have an effect.

Re: Expanding on what we missed with sycophancy

#189

Post by nyarlathotep_ »

neuroelectron wrote: post #168
photonthug wrote: post #150

Earlier quoted context omitted.

> But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case Seems like the same issue as the evil vector [1] and it could have been predicted that this would happen. > It's kind of a wild sign of the times to see a tech company issue this kind of post mortem about a flaw in its te…

Well, that's always what LLM-based AI has been. It can be incredibly convincing but the bottom line is it's just flavoring past text patterns, billions of them it's been "trained" on, which is more accurately described as compressed efficiently onto latent space. Like if someone lived for 10,000 years engaging in small talk at the bar, has heard it all, and just kind of mindlessly and intuitively replied with somethi…

> it's been 4 years now of this and AI still tells me the most obviously wrong nonsense every day.

It's remarkable seeing the change in sentiment in these parts, considering even just a year ago a large part of this forum seemed to regularly proclaim that programmers were done, lawyers were gone in 5 years, "Aye Gee Eye is coming", etc etc.

Post reply on HN