Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Expanding on what we missed with sycophancy

openai.com

Post reply on HN
Print view View original

101–110 of 297 posts

Re: Expanding on what we missed with sycophancy

#101

Post by kergonath »

taurath wrote: post #31
cjbgkagh wrote: post #23

Earlier quoted context omitted.

For many people ChatGPT is already the smartest relationship they have in their lives, not sure how long we have until it’s the most fulfilling. On the upside it is plausible that ChatGPT can get to a state where it can act as a good therapist and help helpless who otherwise would not get help. I am more regularly finding myself in discussions where the other person believes they’re right because they have ChatGPT in…

I think that most smart people underestimate the complexity of fields they aren’t in. ChatGPT may be able to replace a psychology listicle, but it has no affect or ability to read, respond, and intervene or redirect like a human can.

> I think that most smart people underestimate the complexity of fields they aren’t in.

And people deep in new technologies overestimate the potential effect of $new_tech_du_jour. You cannot solve a problem without understanding it and its reasons. And LLMs are not able to understand something.

Re: Expanding on what we missed with sycophancy

#102

Post by codr7 »

low_tech_love wrote: post #74
labrador wrote: post #3

OpenAI mentions the new memory features as a partial cause. My theory as a imperative/functional programmer is that those features added global state to prompts that didn't have it before leading to unpredictability and instabilty. Prompts went from stateless to stateful. As GPT 4o put it: 1. State introduces non-determinism across sessions 2. Memory + sycophancy is a feedback loop 3. Memory acts as a shadow prompt m…

I love the fact that you use its own description to explain what it is, as if it was the expert on itself. I personally cannot see how its own output can be seen as accurate at this level of meta-discussion.

A sign of times to come if you ask me, once it predominantly consumes its own output we're fucked.

Re: Expanding on what we missed with sycophancy

#103

Post by kergonath »

philwelch wrote: post #65
taurath wrote: post #31

Earlier quoted context omitted.

I think that most smart people underestimate the complexity of fields they aren’t in. ChatGPT may be able to replace a psychology listicle, but it has no affect or ability to read, respond, and intervene or redirect like a human can.

You’re comparing ChatGPT to an idealized example of a good human therapist when many actual therapists are either useless or even actively harmful to the mental health of their clients.

But then, the fact that harmful therapist exist is not an excuse to make it worse. It’s an excuse to improve regulations.

“Car accidents happen regardless of what we do, so YOLO and remove safety standards” is never going to fly.

Re: Expanding on what we missed with sycophancy

#104

Post by cjbgkagh »

svieira wrote: post #88
cjbgkagh wrote: post #62

Earlier quoted context omitted.

If you are suggesting that people shouldn't underestimate the difficulty of the jobs of others - my answer is a strong yes. People should strive for accuracy in all cases. But I did suggest that even if true it does not negate my assertion so I am failing to see the relevance. Perhaps I have misunderstood your point.

Sorry, I was rather obscure - you said "My estimations don’t come from my assumption that other people’s jobs are easy, they come from doing applied research in behavioral analytics on mountains of data in rather large data centers." And so I considered the preceding discussion in light of your last sentence. Which makes it sound like you are saying "I've observed the behavior of people and they're often flawed and f…

It wasn't that I observed them being foolish but many behaviors are subtly linked to intelligence and can be combined to create a proxy IQ. It also helps when people search their SAT scores. I noted that the people I typically interact with are much higher IQ than I had expected which incorrectly skewed my believe of the average higher. I noticed that other high IQ individuals were making the same assumptions. I had very much underestimated how little I interact with regular people.

I think we're already finding out that people are doing better with ChatGPT than with their peers, not all peers are created equal, and they can ask ChatGPT things that they cannot ask their peers. I think this trend will continue to the point that most people will prefer discussing things with ChatGPT than with their peers. Given what I know I predict this is a choice many people will make, I'm not passing judgment on that, it's a choice I've also made and I'm fortunate enough to have better peers than most.

Re: Expanding on what we missed with sycophancy

#105

Post by kergonath »

kadushka wrote: post #73
Henchman21 wrote: post #67

Earlier quoted context omitted.

> I think the broader issue here is people using ChatGPT as their own personal therapist. An aside, but: This leads me right to “why do so very many people need therapy?” followed by “why can’t anyone find (or possibly afford) a therapist?” What has gone so wrong for humanity that nearly everyone seems to at least want a therapist? Or is it just the zeitgeist and this is what the herd has decided?

I've never ever thought about needing a therapist. Don't remember anyone in my circle who had ever mentioned it. Similar to how I don't remember anyone going to a palm reader. I'm not trying to diss either profession, I'm sure someone benefits from them, it's just not for me. And I'm sure I'm pretty average in terms of emotional intelligence or psychological issues. Who are all those people who need professional ther…

> I've never ever thought about needing a therapist.

Most people don’t need a therapist. But unfortunately, most people need someone empathic they can talk to and who understands them. Modern life is very short on this sort of people, so therapists have to do.

Re: Expanding on what we missed with sycophancy

#106

Post by didericis »

Jonovono wrote: post #84
NoboruWataya wrote: post #17

I found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the…

It has already replaced therapists, the future is just not evenly distributed yet. There are videos with millions of views on tiktok and comments with hundreds of thousands of likes of teenage girls saying they have gotten more out of 1 week using ChatGPT as a therapist than years of human therapy. Available anytime, cheaper, no judgement, doesn't bring there own baggage, etc.

> no judgement

The value of a good therapist is having an empathetic third party to help you make good judgements about your life and learn how to negotiate your needs within a wider social context.

Depending on the needs people are trying to get met and how bad the people around them are, a little bit of a self directed chatbot validation session might help them feel less beat down by life and do something genuinely positive. So I’m not necessarily opposed to what people are doing with them/in some cases it doesn’t seem that bad.

But calling that therapy is both an insult to genuinely good therapists and dangerous to people with genuine mental/emotional confusion or dysregulation that want help. Anyone with a genuinely pathological mental state is virtually guaranteed to end up deeper in whatever pathology they’re currently in through self directed conversations with chatbots.

Re: Expanding on what we missed with sycophancy

#108

Post by kergonath »

Henchman21 wrote: post #94
kadushka wrote: post #73

Earlier quoted context omitted.

I've never ever thought about needing a therapist. Don't remember anyone in my circle who had ever mentioned it. Similar to how I don't remember anyone going to a palm reader. I'm not trying to diss either profession, I'm sure someone benefits from them, it's just not for me. And I'm sure I'm pretty average in terms of emotional intelligence or psychological issues. Who are all those people who need professional ther…

Well, in my circles its an assumption you’re in therapy. Perhaps this says way more about the circles I’m in that anything else? I was pushed into therapy when I was 12 — which was definitely an exception at the time (1987). As the years have passed therapy has become much much more acceptable. It wouldn’t shock me to learn my own perception is shaped by my experiences; hard to put aside a PoV once acquired.

> Well, in my circles its an assumption you’re in therapy. Perhaps this says way more about the circles I’m in that anything else?

This sounds like an old Woody Allen movie. I don’t want to offend you but it is fascinating. What kind of social circles is it?

In mine, therapy is in general something you do when it’s obvious it’s too late and you are falling in the well of depression and that you try to hide as much as you can.

Re: Expanding on what we missed with sycophancy

#109

Post by autoexec »

Jonovono wrote: post #84
NoboruWataya wrote: post #17

I found the recent sycophancy a bit annoying when trying to diagnose and solve coding problems. First it would waste time praising your intelligence for asking the question before getting to the answer. But more annoyingly if I asked "I am encountering X issue, could Y be the cause" or "could Y be a solution", the response would nearly always be "yes, exactly, it's Y" even when it wasn't the case. I guess part of the…

It has already replaced therapists, the future is just not evenly distributed yet. There are videos with millions of views on tiktok and comments with hundreds of thousands of likes of teenage girls saying they have gotten more out of 1 week using ChatGPT as a therapist than years of human therapy. Available anytime, cheaper, no judgement, doesn't bring there own baggage, etc.

> There are videos with millions of views on tiktok and comments with hundreds of thousands of likes of teenage girls saying they have gotten more out of 1 week using ChatGPT as a therapist than years of human therapy.

You can find influencers on tiktok recommending all kinds of terrible ideas and getting thousands of likes. That's not a very reliable metric. I wouldn't put a lot of faith in a teenage girl's assessment of AI therapy after just one week either, and I certainly wouldn't use that assessment to judge the comparative effectiveness of all human therapists.

I'd also expect ChatGPT to build profiles on people who use it, to use the insights and inferences from that collected data against the user in various ways, to sell that data in some form to third parties, to hand that data over to the state, to hallucinate wildly and unpredictably, and to outright manipulate/censor AI's responses according to ChatGPT's own values and biases or those of anyone willing to pay them enough money.

It's a lot easier to pay a large amount of money to ChatGPT so that the AI will tell millions of vulnerable teenage girls that your product is the solution to their exact psychological problems than it is to pay large amounts of money to several million licensed therapists scattered around the globe.

Maybe you think that ChatGPT is unfailingly ethical in all ways and would never do any of those things, but there are far more examples of companies who abandoned any commitment to ethics they might have started with than there are companies who never got once greedy enough to do those types of things and never ever got bought up by someone who was. I suppose you'd also have to think they'll never have a security breach that would expose the very private information being shared and collected.

Handing over your highly sensitive and very personal medical data to the unlicensed and undependable AI of a company that is only looking for profit seems extremely careless. There are already examples of suicides being attributed to people seeking "therapy" from AI, which has occasionally involved that AI outright telling people to kill themselves. I won't deny that the technology has the potential to do some good things, but every indication is that replacing licensed therapists with spilling all your secrets to a corporate owned and operated AI will ultimately lead to harm.

Re: Expanding on what we missed with sycophancy

#110

Post by cjbgkagh »

codr7 wrote: post #99
alickz wrote: post #92

Earlier quoted context omitted.

>I think most smart people overestimate the intelligence of others for a variety of reasons so they overestimate what it would take for a LLM to beat the output of an average person. I think most people also _vastly_ overestimate how much positive attention the average person gets in their lives It wouldn't surprise me if, for most people, ChatGPT offers them more empathy and understanding than _anyone_ else _ever ha…

It's a pretty sucky solution to that problem imo, and I can see a substantial risk that it causes people to withdraw even more from real relations.

One concern that I do worry about is if LLMs are able to present an false attractive view of the world that the user will become increasingly dependent on the LLMs to maintain that view. A cult of 1. Reminds me of the episode 'Safe Space' from South Park but instead of Butters filtering content it'll be the LLM. People are already divorced enough from reality - but I see no reason why they couldn't be more divorced, at least temporarily.
Post reply on HN