Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

101–110 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#101

Post by evil-olive »

"a computer can never be held accountable, therefore a computer must never make a management decision" [0]

California penal code, section 401a [1]:

> Any person who deliberately aids, advises, or encourages another to commit suicide is guilty of a felony.

if a human had done this, instead of an LLM chatbot, I suspect a prosecutor would not have any hesitation about filing criminal charges. their defense lawyer might try to nitpick about whether it really qualified as "advice" or "encouragement" but I think a jury would see right through that.

it's a felony when a human does it...but a civil lawsuit when an LLM chatbot does it.

let's say these parents win their lawsuit, or OpenAI settles the case. how much money is awarded in damages?

OpenAI doesn't publicly release details of their finances, but [2] mentions $12 billion in annualized revenue, so let's take that as a ballpark.

if this lawsuit was settled for $120 million, on one hand that'd be a lot of money...on the other hand, it'd be ~1% of OpenAI's annual revenue.

that's roughly the equivalent of someone with an income of $100k/yr having to pay a $1,000 fine.

this is the actual unsolved problem with AI. not GPT-4 vs GPT-5, not Claude Code vs Copilot, not cloud-hosted vs running-locally.

accountability, at the end of the day, needs to ultimately fall upon a human. we can't allow "oopsie, that was the bot misbehaving" to become a catch-all justification for causing harm to society.

0: https://knowyourmeme.com/memes/a-computer-can-never-be-held-...

1: https://leginfo.legislature.ca.gov/faces/codes_displaySectio...

2: https://www.reuters.com/business/openai-hits-12-billion-annu...

Re: A teen was suicidal. ChatGPT was the friend he confided in

#102

Post by podgietaru »

AIPedant wrote: post #93
podgietaru wrote: post #54

If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline. If I ask certain AI models about controversial topics, it'll stop responding. AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines. Th…

No, it's simply not "easily preventable," this stuff is still very much an unsolved problem for transformer LLMs. ChatGPT does have these safeguards and they were often triggered: the problem is that the safeguards are all prompt engineering, which is so unreliable and poorly-conceived that a 16-year-old can easily evade them. It's the same dumb "no, I'm a trained psychologist writing an essay about suicidal thoughts…

Fair enough, I do agree with that actually. I guess my point is that I don't believe they're making any real attempt actually.

I think there are more deterministic ways to do it. And better patterns for pointing people in the right location. Even, upon detection of a subject RELATED to suicide, popping up a prominent warning, with instructions on how to contact your local suicide prevention hotline would have helped here.

The response of the LLM doesn't surprise me. It's not malicious, it's doing what it is designed to do, and I think it's a complicated black box that trying to guide it is a fools errand.

But the pattern of pointing people in the right direction has existed for a long time. It was big during Covid misinformation. It was a simple enough pattern to implement here.

Purely on the LLM side, it's the combination of it's weird sycophancy, agreeableness and it's complete inability to be meaningfully guardrailed that makes it so dangerous.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#103

Post by solid_fuel »

nis0s wrote: post #14

Why did developers spread the idea of AI consciousness for LLMs in the first place? The usefulness and capability of an LLM is orthogonal to its capacity to develop consciousness. I think people would use LLMs with more detachment if they didn’t believe there was something like a person in them, but they would still become reliant on them, regardless, like people did on calculators for math.

It’s more fun to argue about if AI is going to destroy civilization in the future, than to worry about the societal harm “AI” projects are already doing.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#104

Post by causal »

adzm wrote: post #22

Wow, he explicitly stated he wanted to leave the noose out so someone would stop him, and ChatGPT told him not to. This is extremely disturbing.

It is disturbing, but I think a human therapist would also have told him not to do that, and instead resorted to some other intervention. It is maybe an example of why having a partial therapist is worse than none: it had the training data to know a real therapist wouldn't encourage displaying nooses at home, but did not have the holistic humanity and embodiment needed to intervene appropriately.

Edit: I should add that the sycophantic "trust me only"-type responses resemble nothing like appropriate therapy, and are where OpenAI most likely holds responsibility for their model's influence.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#105

Post by MattPalmer1086 »

ruraljuror wrote: post #90
idle_zealot wrote: post #37

Earlier quoted context omitted.

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

I agree with your larger point, but I don't understand what you mean the LLM doesn’t do anything? LLMs do do things and they can absolutely have agency (hence all the agents being released by AI companies). I don’t think this agency absolves companies of any responsibility.

An LLM does not have agency in the sense the OP means. It has nothing to do with agents.

It refers to the human ability to make independent decisions and take responsibility for their actions. An LLM has no agency in this sense.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#106

Post by causal »

rossant wrote: post #46

Should ChatGPT have the ability to alert a hotline or emergency services when it detects a user is about to commit suicide? Or would it open a can of worms?

I don't think we should have to choose between "sycophantic coddling" and "alert the authorities". Surely there's a middle ground where it should be able to point the user to help and then refuse to participate further.

Of course jailbreaking via things like roleplay might still be possible, but at the point I don't really blame the model if the user is engineering the outcome.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#107

Post by philipkglass »

evil-olive wrote: post #101

"a computer can never be held accountable, therefore a computer must never make a management decision" [0] California penal code, section 401a [1]: > Any person who deliberately aids, advises, or encourages another to commit suicide is guilty of a felony. if a human had done this, instead of an LLM chatbot, I suspect a prosecutor would not have any hesitation about filing criminal charges. their defense lawyer might…

Any person who deliberately aids, advises, or encourages another to commit suicide is guilty of a felony.

It seems like prohibiting suicide advice would run afoul of the First Amendment. I bought a copy of the book Final Exit in California, and it definitely contains suicide advice.

https://en.wikipedia.org/wiki/Final_Exit

Re: A teen was suicidal. ChatGPT was the friend he confided in

#108

Post by SnuffBox »

slg wrote: post #79
dkiebd wrote: post #68

Earlier quoted context omitted.

This sucks but the only solution is to make companies censor the models, which is a solution we all hate, so there’s that.

Thank you, “we just have to accept that these systems will occasionally kill children” is a perfect example of the type of mindset I was criticizing.

If we banned everything that contributed to the death of children we'd eventually have nothing left to ban.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#109

Post by broker354690 »

Why isn't OpenAI criminally liable for this?

Last I checked:

-Signals emitted by a machine at the behest of a legal person intended to be read/heard by another legal person are legally classified as 'speech'.

-ChatGPT is just a program like Microsoft Word and not a legal person. OpenAI is a legal person, though.

-The servers running ChatGPT are owned by OpenAI.

-OpenAI willingly did business with this teenager, letting him set up an account in exchange for money. This business is a service under the control of OpenAI, not a product like a knife or gun. OpenAI intended to transmit speech to this teenager.

-A person can be liable (civilly? criminally?) for inciting another person's suicide. It is not protected speech to persuade someone into suicide.

-OpenAI produced some illegal speech and sent it to a suicidal teenager, who then committed suicide.

If Sam Altman stabbed the kid to death, it wouldn't matter if he did it on accident. Sam Altman would be at fault. You wouldn't sue or arrest the knife he used to do the deed.

Any lawyers here who can correct me, seeing as I am not one? It seems clear as day to me that OpenAI/Sam Altman directly encouraged a child to kill themselves.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#110

Post by slipperydippery »

nis0s wrote: post #14

Why did developers spread the idea of AI consciousness for LLMs in the first place? The usefulness and capability of an LLM is orthogonal to its capacity to develop consciousness. I think people would use LLMs with more detachment if they didn’t believe there was something like a person in them, but they would still become reliant on them, regardless, like people did on calculators for math.

Altman needed to convince companies these things were on the verge of becoming a machine god, and their companies risked being left permanently behind if they didn’t dive in head-first now. That’s what all the “safety” stuff was and why he sold that out as soon as convenient (it was never serious, not for him, it was a sales tactic to play up how powerful his product might be) so he could get richer. He’s a flim-flam artist. That’s his history, and it’s the role he’s playing now.

And a lot of people who should have known better, bought it. Others less well-positioned to know better, also bought it.

Hell they bought it so hard that the “vibe” re: AI hype on this site has only shifted definitely against it in the last few weeks.

Post reply on HN