Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI says over a million people talk to ChatGPT about suicide weekly

techcrunch.com

Post reply on HN
Print view View original

471–480 of 585 posts

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#471

Post by danaris »

fragmede wrote: post #361
simonklitj wrote: post #313

Earlier quoted context omitted.

Thank you! On top of that, it’s hard to measure “potential suicides averted,” and comparing that with “actual suicides caused/assisted with” would be incommnsurable. And working to set a threshold for what we would consider acceptable? No thanks

Real life trolly problem! If you pull the lever, some people on this track will die (by sucide). If you don't pull the lever, some people will still die from suicide. By not pulling the lever, and simply banning discussion of suicide entirely, your company gets to avoid a huge PR disaster, and you get more money because line go up. If you pull the lever and let people talk about suicide on your platform, you may avoi…

....but if you pull the lever and let people talk about suicide on your platform, your platform will actively contribute to some unknowable number of suicides.

There is, at this time, no way to determine how the number it would contribute to would compare to the number it would prevent.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#472

Post by freedomben »

ml-anon wrote: post #411
josephg wrote: post #341

Earlier quoted context omitted.

> Most people who have worked with a professional therapist understand intuitively why the only helpful feedback from an LLM to someone who needs professional help is: get professional help. I'm not a therapist, but as I understand it most therapy isn't about suicide, and doesn't carry suicide risk. Most therapy is talking through problems, and helping the patient rewrite old memories and old beliefs using more helpf…

“If it is prompted well” What the fuck does this even mean? How do you test or ensure it. Because based on the actual outcomes ChatGPT is 0-1 for preventing suicides (going as far as to outright encourage one).

If you're going to make the sample size one, and use the most egregious example, you make pretty much anything that has ever been born or built look terrible. Given there are millions of people using chat, GPT and others for therapy every week, maybe even everyday, citing a record of being 0-1 is pretty ridiculous.

To be clear, I'm not defending this particular case. Chat GPT clearly messed up bad.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#473

Post by wat10000 »

Al-Khwarizmi wrote: post #236
probably_wrong wrote: post #229

If you haven't read the article (or even if you have but didn't click on outgoing links twice ) the NYT story about how ChatGPT convinced a suicidal teen not to look for help [1] should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Here's ChatGPT discouraging said teenager from asking for help: > “I want to leave my noose in my room so someone finds it and tries to stop me…

We would need the big picture, though... maybe it caused that death (which is awful) but it's also saving lives? If there are that many people confiding in it, I wouldn't be surprised if it actually prevents some suicides with encouraging comments, and that's not going to make the news. Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the…

Our standard approach for new medical treatments is to require proof of safety and efficacy before it's made available to the general public. This is because it's very, very easy for promising-looking treatments to end up being harmful.

"Before declaring that it shouldn't be near anyone with psychological issues" is backwards. Before providing it to people with psychological issues, someone should study whether the positive impact is greater than the negative.

Trouble is, this is such a generalized tool that it's very hard to do that.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#474

Post by GuinansEyebrows »

raducu wrote: post #435
alex-moon wrote: post #317

Earlier quoted context omitted.

I suspect you've never done therapy yourself. Most people who have worked with a professional therapist understand intuitively why the only helpful feedback from an LLM to someone who needs professional help is: get professional help. AIs are really good at doing something to about 80%. When the stakes are life or death, as they are with someone who is suicidal, that is a good example of a time when 80% isn't good en…

> I suspect you've never done therapy yourself. Most people who have worked with a professional therapist understand intuitively why the only helpful feedback from an LLM to someone who needs professional help is: get professional help. AIs are really good at doing something to about 80%. I'm shocked that GPT-5 or Gemini can code so well, yet if I paste a 30 line (heated) chat conversation between my wife and I it me…

> if I paste a 30 line (heated) chat conversation between my wife and I

i can't imagine how violated i would feel if i found out my partner was sending our private conversations to a nonprivate LLM chatbot. it's not a friend with a sense of care; it's a text box whose contents are ingested by a corporation with a vested interest in worsening communication between humans. scary stuff.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#475

Post by TZubiri »

Damn if you do something, damn if you don't.

I think that the approach and advantage of CA/US companies is to be bold and do shit ("you can just do things"/"move fast break things"), they consciously manage huge legal liabilities (which are not minor in the US), I don't know how they manage to stay afloat, probably tight legal teams, and enough revenue to offset the liabilities.

But the scope of ChatGPT is one of the biggest I've seen so far, by default it encompasses everything, whatever is out of scope it's because they specifically blacklist it, and even then they keep on dishing out legal, medical advice, and psychiatric advice.

I think one of the systemic risks is a legal liability crisis, not just for chatgpt, but for the whole US Tech market and therefore the stock market (almost all top stocks are tech). Like if you start thinking what will the next 2008 would be, I think legal liabilities are up there, along with nuclear energy snafus and war.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#476

Post by malka1986 »

alex-moon wrote: post #396
ethbr1 wrote: post #333

Earlier quoted context omitted.

You're holding up a perfect status quo that doesn't correspond to reality. Countries vary, but in the US and many places there's a shortage of quality therapists. Thus for many people the actual options are {no therapy} and {LLM therapy}. > This is why we have safety regulations, why we don't let people drive cars that haven't passed tests, build with materials that haven't passed tests, eat food that hasn't passed t…

I don't accept the unregulated and uncontrolled use of LLMs for therapy for the same reason I don't accept arguments like "We should deregulate food safety because it means more food at lower cost to consumers" or "We should waive minimum space standards for permitted developments on existing buildings because it means more housing." We could "solve" the homeless problem tomorrow simply by building tenements (that is…

> I don't accept the unregulated and uncontrolled use of LLMs for therapy for the same reason I don't accept arguments like "We should deregulate food safety because it means more food at lower cost to consumers"

That is not the argument. The argument is not about 'lower cost', it is about availability. There are not enough shrinks for everyone who would need it.

So it would be "We should deregulate food safety to avoid starving", which would be a valid argument.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#477

Post by TZubiri »

probably_wrong wrote: post #229

If you haven't read the article (or even if you have but didn't click on outgoing links twice ) the NYT story about how ChatGPT convinced a suicidal teen not to look for help [1] should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Here's ChatGPT discouraging said teenager from asking for help: > “I want to leave my noose in my room so someone finds it and tries to stop me…

I know that minors under age 13 are not allowed to use the app. But 13-18 is fine? Not sure why. Might also be worth looking into making apps like these 18+. Whether by law or by liability, if someone 20+ gets, say, food poisoning by getting recipes from chatgpt, then you can argue that it's the user's fault for not fact checking, but if a 15yo kid gets food poisoning, it's harder to argue that it's the kid's fault.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#478

Post by DoneWithAllThat »

itchyjunk wrote: post #336

It seems like people here have already made up their mind about how bad llms are. So just my anecdote here, it helped me out of some really dark places. Talking to humans (non psychologists) had the opposite effect. Between a non professional and an llm, i'd pick llm for myself. Others should definitely seek help.

Given how ideologically captured the therapist industry is now I think it’s very hard to say that an LLM do such things is objectively worse.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#479

Post by wat10000 »

cal85 wrote: post #460
toofy wrote: post #325

Earlier quoted context omitted.

people are overcomplicating this, the big picture is simple af: if a therapist was ever found to have said to this to a suicidal person, they would be immediately stripped of their license and maybe jailed.

True. But it feels like a fairer comparison would be with a huge healthcare company that failed to vet one of its therapists properly, so a crazy pro-suicide therapist slipped through the net. Would we petition to shut down the whole company for this rare event? I suppose it would depend on whether the company could demonstrate what it is doing to ensure it doesn’t happen again.

Maybe you shouldn't shut down OpenAI over this. But each instance of a particular ChatGPT model is the same as all the others. This is like a company that has a magical superhuman therapist that can see a million patients a day. If they're found to be encouraging suicide, then they need to be stopped from providing therapy. The fact that this is the company's only source of revenue might mean that the company has to shut down over this, but that's just a consequence of putting all your eggs in one basket.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#480

Post by [deleted] »

formerly_proven wrote: post #351
mschuster91 wrote: post #335

Earlier quoted context omitted.

> If the client needs to be reminded to box breathe and that they're using all or nothing thinking again to get them off of the ledge, does that really require a human who's only available once a week to gently remind them of that when the therapist isn't going to be available for four more days and ChatGPT's available right now? There are 24/7 suicide prevention hotlines at least in many countries in Europe as well…

> There are 24/7 suicide prevention hotlines at least in many countries in Europe as well as US states. My understanding is these will generally just send the cops after you if the operator concludes you are actually suicidal and not just looking for someone to talk to for free.

[deleted]
Post reply on HN