Expanding on what we missed with sycophancy • Hacker News Board

Re: Expanding on what we missed with sycophancy

#221

Post by bluefirebrand » Sat, May 03, 2025, 5:56 AM UTC

pixl97 wrote: post #125

kergonath wrote: post #101

Earlier quoted context omitted.

> I think that most smart people underestimate the complexity of fields they aren’t in. And people deep in new technologies overestimate the potential effect of $new_tech_du_jour. You cannot solve a problem without understanding it and its reasons. And LLMs are not able to understand something.

People have solved problems for most of history without understanding them. For example problems can be brute forced.

You cannot brute force a problem without understanding the problem

I think you have mistaken "understanding the problem" with "knowing a solution to the problem"

Re: Expanding on what we missed with sycophancy

#222

Post by drdeca » Sat, May 03, 2025, 5:57 AM UTC

danenania wrote: post #156

codr7 wrote: post #154

Earlier quoted context omitted.

Reality as in the real world, it is what it is, no one decides.

We're talking about psychology, therapy, sycophancy, etc. None of this is empirical. If someone thinks they can, say, create a billion dollar startup, whether they can really do it or not is a subjective determination. The AI might tell the person they can do it. You might tell them they can't, that the AI is sycophantic, and that they should stop talking to it because they're losing touch with reality. But is the AI…

That depends on whether they are capable of creating a billion dollar startup.

If they aren’t, and I say they aren’t, then I am correct. If they are, and the AI’s output says they are, then AI’s output is correct.

Re: Expanding on what we missed with sycophancy

#223

Post by istjohn » Sat, May 03, 2025, 6:10 AM UTC

didericis wrote: post #106

Jonovono wrote: post #84

Earlier quoted context omitted.

It has already replaced therapists, the future is just not evenly distributed yet. There are videos with millions of views on tiktok and comments with hundreds of thousands of likes of teenage girls saying they have gotten more out of 1 week using ChatGPT as a therapist than years of human therapy. Available anytime, cheaper, no judgement, doesn't bring there own baggage, etc.

> no judgement The value of a good therapist is having an empathetic third party to help you make good judgements about your life and learn how to negotiate your needs within a wider social context. Depending on the needs people are trying to get met and how bad the people around them are, a little bit of a self directed chatbot validation session might help them feel less beat down by life and do something genuinely…

I wouldn't trust ChatGPT to help someone in a mental health crisis, but I would be glad to find out my dad had started using Claude Sonnet to process his transition into retirement. I believe Sonnet would encourage a user to seek professional help when appropriate, too. In my experience, genuinely good therapists are hard to find--probably 75% of them are going to be strictly worse than Sonnet.

Re: Expanding on what we missed with sycophancy

#224

Post by istjohn » Sat, May 03, 2025, 6:11 AM UTC

mynameisash wrote: post #185

Spooky23 wrote: post #169

Earlier quoted context omitted.

The problem with this is they are practicing like medical providers without any quality assurance or controls to ensure they are behaving appropriately. Therapy is already a bit of grey zone… you can have anyone from a psychologist, a social worker, an untrained deacon, etc “counseling” you. This is worse. Hell, I’ve been a coach in different settings - players will ask for advice about all sorts of things. There’s a…

Kind of reminds me of an interview question that a friend of mine suggested for when I conduct interviews: Pick your favorite/strongest language. How would you rate yourself, where 0 is "complete newbie" and 10 is "I invented the language"? My friend, an EXTREMELY competent C++ programmer, rates himself 4/10 because he knows what he doesn't know. I've interviewed people who rated themselves 9 or 10/10 but couldn't re…

Sounds like a bad question then, no?

Re: Expanding on what we missed with sycophancy

#225

Post by motoxpro » Sat, May 03, 2025, 6:19 AM UTC

The wild thing was this is what people "wanted"

- The agreeableness came from heavier weighted user signals, not from OpenAI. The users give better feedback when the answers positively respond to their questions.

- The most common use case for AI is a therapist.

- The reward signals (user feedback) being positive means that the response helped them emotionally. Which is easy to imagine someone asking a hard question "Am I enough in this situation?" The model then assures that the user IS enough, and the user gives it a thumbs up.

- I would say: 1. Coding is not the killer use case. Productivity is not the killer use case. The killer use case is emotional support. 2. Who is anyone to say that this use case is wrong? 3. It is impossible to say what is "right" in this situation. (everyone thinks their advice to a friend is the right advice, even when it is opposite). So there is no objectivity.

- I think a lot of people who wanted an "open, do anything" model, are going to have more problems and dissonance with this idea than they would about the model allowing people to create weapons, allow self-harm, etc. because it will feel uncomfortable that a model is shaping reality for a large portion of the population.

https://hbr.org/2025/04/how-people-are-really-using-gen-ai-i...

Re: Expanding on what we missed with sycophancy

#226

Post by istjohn » Sat, May 03, 2025, 6:38 AM UTC

kadushka wrote: post #73

Henchman21 wrote: post #67

Earlier quoted context omitted.

> I think the broader issue here is people using ChatGPT as their own personal therapist. An aside, but: This leads me right to “why do so very many people need therapy?” followed by “why can’t anyone find (or possibly afford) a therapist?” What has gone so wrong for humanity that nearly everyone seems to at least want a therapist? Or is it just the zeitgeist and this is what the herd has decided?

I've never ever thought about needing a therapist. Don't remember anyone in my circle who had ever mentioned it. Similar to how I don't remember anyone going to a palm reader. I'm not trying to diss either profession, I'm sure someone benefits from them, it's just not for me. And I'm sure I'm pretty average in terms of emotional intelligence or psychological issues. Who are all those people who need professional ther…

You saying talk therapy just isn't for you is like me saying physical therapy just isn't for me. Well, I'm glad that everything is functioning well for you in that area, but it would be kind of silly for me to write off physical therapy because I haven't needed it yet. Talk therapy can help people with social anxiety or chronic depression, but it can also help people cope with major life changes like losing a loved one, losing a job, getting divorced, or even retiring. We all have difficult moments in our life.

Re: Expanding on what we missed with sycophancy

#227

Post by thinkingemote » Sat, May 03, 2025, 6:42 AM UTC

bredren wrote: post #201

The strangest thing I noticed during this model period was that that the AI suggested we keep an inside joke together. I had a dictation error on a message I sent, and when it repeated the text later I asked what it was talking about. It was able to point at my message and guess that maybe it was a mistake. When I validated that and corrected it, the AI thought it would be a cute/funny joke for us to keep together. I…

Perhaps the "rewards" the system receives has lead to a reward mechanism where the system tries to not get bad reports from the user. Hiding bad performance as a secret joke would fit in with that

Re: Expanding on what we missed with sycophancy

#228

Post by low_tech_love » Sat, May 03, 2025, 6:44 AM UTC

codr7 wrote: post #102

low_tech_love wrote: post #74

Earlier quoted context omitted.

I love the fact that you use its own description to explain what it is, as if it was the expert on itself. I personally cannot see how its own output can be seen as accurate at this level of meta-discussion.

A sign of times to come if you ask me, once it predominantly consumes its own output we're fucked.

I still hope there is a future where the slop becomes so blatant that the majority (or at least a good portion) of the users lose interest, or something like that. The world is harder to predict than our brain wants us to think (at least I hope so). The more I think about AI the more it sounds like the problem is that companies wanted to put out whatever random crap they had cooking as quickly as possible just to try to win some race, but we have still not converged to the actual real, paradigm-changing applications. And I’m not sure that the answer is in the big corps because for them maybe it’s easier/more profitable to simply keep giving people what they want instead of actual useful things.

Re: Expanding on what we missed with sycophancy

#229

Post by lblume » Sat, May 03, 2025, 6:45 AM UTC

motoxpro wrote: post #225

The wild thing was this is what people "wanted" - The agreeableness came from heavier weighted user signals, not from OpenAI. The users give better feedback when the answers positively respond to their questions. - The most common use case for AI is a therapist. - The reward signals (user feedback) being positive means that the response helped them emotionally. Which is easy to imagine someone asking a hard question…

> The most common use case for AI is a therapist.

Source? Sounds like a fairly wild claim.

Re: Expanding on what we missed with sycophancy

#230

Post by istjohn » Sat, May 03, 2025, 6:54 AM UTC

kergonath wrote: post #105

kadushka wrote: post #73

Earlier quoted context omitted.

I've never ever thought about needing a therapist. Don't remember anyone in my circle who had ever mentioned it. Similar to how I don't remember anyone going to a palm reader. I'm not trying to diss either profession, I'm sure someone benefits from them, it's just not for me. And I'm sure I'm pretty average in terms of emotional intelligence or psychological issues. Who are all those people who need professional ther…

> I've never ever thought about needing a therapist. Most people don’t need a therapist. But unfortunately, most people need someone empathic they can talk to and who understands them. Modern life is very short on this sort of people, so therapists have to do.

I disagree. Most people will go through a time in their life where they could benefit from a therapist--maybe to cope with bullying in high school--perhaps to mentally and emotionally process a terminal illness or the loss of a spouse or child.

And then some people will benefit from just having someone to talk to about the stresses of daily life. I don't think that means they have an inadequate social network or that it reflects poorly on modern society. We should be happy to live in a time where we have the resources and ability to proactively care for our mental health just as we do our teeth.