OpenAI says over a million people talk to ChatGPT about suicide weekly • Hacker News Board

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#431

Post by cindyllm » Tue, Oct 28, 2025, 1:39 PM UTC

exasperaited wrote: post #355

imiric wrote: post #291

Earlier quoted context omitted.

But that's not the issue. The issue is that a kid is talking to a machine without supervision in the first place, and presumably taking advice from it. The main questions are: where are the guardians of this child? What is the family situation and living environment? A child thinking about suicide is clearly a sign that there are far greater problems in their life than taking advice from a machine. Let's address thos…

> But that's not the issue. It is the issue at least in the sense that it's the one I was personally responding to, thanks. And there are many issues, not just the one you are choosing to focus on. "Deeper societal problems" is a typical get-out clause for all harmful technology. It's not good enough. Like, in the USA they say "deeper societal problems" about guns; other countries ban them and have radically fewer gu…

[dead]

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#432

Post by TiredOfLife » Tue, Oct 28, 2025, 1:42 PM UTC

wonderwonder wrote: post #40

Most people would really benefit from going to the gym. I'm not trying to downplay serious mental illness as its absolutely real. For many though just going to the gym several times a week or another form of serious physical exertion can make a world of difference. Since I started taking the gym seriously again I feel like a new man. Any negative thoughts are simply gone. (The testosterone helps as well) This is comi…

1) are you going to finance that?

2) are you going to make sure other people at the gym don't make fun of me?

>> Besides my wife and kids its almost total isolation

Good old "if you have money trouble try decreasing your caviar and truffle intake to only two meals a day"

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#433

Post by nihzm » Tue, Oct 28, 2025, 1:43 PM UTC

ml-anon wrote: post #404

Al-Khwarizmi wrote: post #236

Earlier quoted context omitted.

We would need the big picture, though... maybe it caused that death (which is awful) but it's also saving lives? If there are that many people confiding in it, I wouldn't be surprised if it actually prevents some suicides with encouraging comments, and that's not going to make the news. Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the…

This entire comment section is full of wide eyed nonsense like this. It’s honestly frightening that we are even humoring this point of view.

Since as you say this utilitarian view is rather common, perhaps it would good to show _why_ this is problematic by presenting a counterargument.

The basic premise under GP's statements is that although not perfect, we should use the technology in such a way that it maximizes the well being of the largest number of people, even if comes at the expense of a few.

But therein lies a problem: we cannot really measure well being (or utility). This becomes obvious if you look at individuals instead of the aggregate: imagine LLM therapy becomes widespread and a famous high profile person and your (not famous) daughter end up in "the few" for which LLM therapy goes terribly wrong and commit suicide. The loss of the famous person will cause thousands (perhaps millions) people to be a bit sad, and the loss of your daughter will cause you unimaginable pain. Which one is greater? Can they even be be compared? And how many people with a successful LLM therapy are enough to compensate for either one?

Unmeasurable well-being then makes these moral calculations at best inexact and at worst completely meaningless. And if they are truly meaningless, how can they inform your LLM therapy policy decisions?

Suppose for the sake of the argument we accept the above, and there is a way to measure well being. Then would it be just? Justice is a fuzzy concept, but imagine we reverse the example above: many people lose their lives because of bad LLM therapy, but one very famous person in the entertainment industry is saved by LLM therapy. Let's suppose then that this famous persons' well being plus the millions of spectators' improved well-being (through their entertainment) is worth enough to compensate the people who died.

This means saving a famous funny person justifies the death of many. This does not feel just, does it?

There is a vast amount of literature on this topic (criticisms of utilitarianism).

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#434

Post by TiredOfLife » Tue, Oct 28, 2025, 1:48 PM UTC

yieldcrv wrote: post #28

I talk to ChatGPT about topics I feel society isnt enlightened enough to talk about I feel suicide is heavily misunderstood as well People just copypasta prevention hotlines and turn their minds off from the topic Although people have identified a subset of the population that is just impulsively considering suicide and can be deterred, it doesnt serve the other unidentified subsets who are underserved by merely dist…

> People just copypasta prevention hotlines and turn their minds off from the topic

But ChatGPT does exactly the same.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#435

Post by raducu » Tue, Oct 28, 2025, 1:51 PM UTC

alex-moon wrote: post #317

Al-Khwarizmi wrote: post #236

Earlier quoted context omitted.

We would need the big picture, though... maybe it caused that death (which is awful) but it's also saving lives? If there are that many people confiding in it, I wouldn't be surprised if it actually prevents some suicides with encouraging comments, and that's not going to make the news. Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the…

I suspect you've never done therapy yourself. Most people who have worked with a professional therapist understand intuitively why the only helpful feedback from an LLM to someone who needs professional help is: get professional help. AIs are really good at doing something to about 80%. When the stakes are life or death, as they are with someone who is suicidal, that is a good example of a time when 80% isn't good en…

> I suspect you've never done therapy yourself. Most people who have worked with a professional therapist understand intuitively why the only helpful feedback from an LLM to someone who needs professional help is: get professional help. AIs are really good at doing something to about 80%.

I'm shocked that GPT-5 or Gemini can code so well, yet if I paste a 30 line (heated) chat conversation between my wife and I it messes up about what 5% of those lines actually mean -- spectacularly so.

It's interesting to ask it it analyze the conversation in various psychotherapeutic frameworks, because I'm not well versed in those and its conclusions are interesting starting points, but it only gets it right about 30% of the time.

All LLMs that I tested are TERRIBLE for actual therapy, because I can make it change its mind in 1-2 lines by adding some extra "facts". I can make it say anything.

LLMs completely lose the plot. They might be good for someone who needs self-validation and a feeling someone is listening, but for actual skill building, they're complete shit as therapists.

I mean, most therapists are complete shit as therapists but that's besides the point.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#436

Post by moralestapia » Tue, Oct 28, 2025, 1:52 PM UTC

JKCalhoun wrote: post #398

moralestapia wrote: post #334

Earlier quoted context omitted.

That's really really bad. But also, how many people has it talked out of doing it? We need the full picture.

That is data that is impossible to get.

>OpenAI says over a million people talk to ChatGPT about suicide weekly

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#437

Post by JKCalhoun » Tue, Oct 28, 2025, 1:56 PM UTC

moralestapia wrote: post #436

JKCalhoun wrote: post #398

Earlier quoted context omitted.

That is data that is impossible to get.

>OpenAI says over a million people talk to ChatGPT about suicide weekly

We don't know how many of those people would have gone through with a suicide but for LLMs.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#438

Post by lagosfractal42 » Tue, Oct 28, 2025, 2:00 PM UTC

jacquesm wrote: post #215

true_religion wrote: post #192

Earlier quoted context omitted.

HIPAA only applies to covered healthcare entitites. If you walk into a McDonalds' and talk about your suicidal ideation with the cashier, that's not HIPAA covered. To become a covered entity, the business has either work with a healhcare provider, health data trasmiter, or do business as one. Notably, even in the above case, HIPAA only applies to the healthcare part of the entity. So if McDonald's collocated pharmaci…

Arguably, if you start giving answers to these kind of questions your chatbot just became a medical device.

What about a medicine book? Is that also a medical device?

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#439

Post by criddell » Tue, Oct 28, 2025, 2:01 PM UTC

nelox wrote: post #416

spookie wrote: post #353

Earlier quoted context omitted.

Ok then, the LLMs must pass the same tests and be as regulated as therapists. After all, it should be easy peasy (:

What tests? The term “therapist” is not protected in most jurisdictions. No regulation required. Almost anyone can call themselves a therapist.

In every state you have to have a license to practice.

The advice to not leave the noose out is likely enough for ChatGPT to lose it's license to practice (if it had one).

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#440

Post by lagosfractal42 » Tue, Oct 28, 2025, 2:03 PM UTC

ml-anon wrote: post #394

Dilettante_ wrote: post #359

Earlier quoted context omitted.

Can you give just a little bit more effort explaining why you say that?

You know, usually it’s positive claims which are supposed to be substantiated, such as the claim that “LLMs can be good at therapy”. Holy shit, this thread is insane.

You added nothing to the thread. Just get out.