Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

41–50 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#41

Post by fzzzy »

nis0s wrote: post #14

Why did developers spread the idea of AI consciousness for LLMs in the first place? The usefulness and capability of an LLM is orthogonal to its capacity to develop consciousness. I think people would use LLMs with more detachment if they didn’t believe there was something like a person in them, but they would still become reliant on them, regardless, like people did on calculators for math.

The Eliza effect is incredibly powerful, regardless of whether developers have spread the idea of AI consciousness or not. I don’t believe people would use LLMs with more detachment if developers had communicated different ideas. The Eliza effect is not new.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#42

Post by Mallowram »

bigyabai wrote: post #17
drewbeck wrote: post #10

Whenever people say that Apple is behind on AI, I think about stories like this. Is this the Siri people want? And if it is easy to prevent, why didn't OpenAI? Some companies actually have a lot to lose if these things go off the rails and can't just 'move fast and break things' when those things are their customers, or the trust their customers have in them. My hope is that OpenAI actually does have a lot to lose; m…

When people tell you that Apple is behind on AI, they mean money. Not AI features, not AI hardware, AI revenue. And Apple is behind on that - they've got the densest silicon in the world and still play second fiddle to Nvidia. Apple GPU designs aren't conducive to non-raster workloads, they fell behind pretty far by obsessing over a less-profitable consumer market. For whatever it's worth, I also hope that OpenAI can…

[dead]

Re: A teen was suicidal. ChatGPT was the friend he confided in

#43

Post by slg »

TheCleric wrote: post #7
TillE wrote: post #5

I would've thought that explicit discussion of suicide is one of those topics that chatbots will absolutely refuse to engage with. Like as soon as people started talking about using LLMs as therapists, it's really easy to see how that can go wrong.

Well everyone seemed to turn on the AI ethicists as cowards a few years ago, so I guess this is what happens.

People got so upset that LLMs wouldn’t say the n-word to prevent a hypothetical nuclear bomb from going off so we now have LLMs that actively encourage teenagers to kill themselves.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#44

Post by kayodelycaon »

Kapura wrote: post #3

this is devestating. reading these messages to and from the computer would radicalize anybody. the fact that the computer would offer a technical analysis of how to tie a noose is damning. openai must be compelled to protect the users when they're clearly looking to harm themselves. it is soulless to believe this is ok.

A noose is really basic information when it comes to tying knots. It’s also situationally useful, so there’s a good reason to include it in any educational material.

The instructions are only a problem in the wrong context.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#45

Post by bigyabai »

Mallowram wrote: post #42
bigyabai wrote: post #17

Earlier quoted context omitted.

When people tell you that Apple is behind on AI, they mean money. Not AI features, not AI hardware, AI revenue. And Apple is behind on that - they've got the densest silicon in the world and still play second fiddle to Nvidia. Apple GPU designs aren't conducive to non-raster workloads, they fell behind pretty far by obsessing over a less-profitable consumer market. For whatever it's worth, I also hope that OpenAI can…

[dead]

If I had a dime for every "CUDA is worthless" comment I've seen since the crypto craze, I could fund the successor to TSMC out of pocket.

Whatever the case is, the raster approach sure isn't winning Apple and AMD any extra market share. Barring any "spherical cow" scenarios, Nvidia won.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#47

Post by rideontime »

idle_zealot wrote: post #37
rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

I completely agree and did not intend to absolve them of their guilt in any way. As far as I see it, this kid's blood is on Sam Altman's hands.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#48

Post by neom »

I've been thinking recently there should probably be a pretty stringent onboarding assessment for these things, something you have to sit through and something that both fully explains what they are and how they work, but also provides an experience that removes the magic from them. I also wish they would deprecate 4o, I know 2 people right now who are currently reliant on it, when they paste me some of the stuff it says... sweeping agreement of wildly inappropriate generalization, I'm sure it's about to end a friends marriage.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#49

Post by rideontime »

MBCook wrote: post #35
throwawaybob420 wrote: post #34

Earlier quoted context omitted.

idk dude if your technology encourages a teenager to kill itself and prevents him from alerting his parents via a cry for help, I don’t care how “beneficial” it is.

I agree. If there was one death for 1 million saves, maybe. Instead, this just came up in my feed: https://arstechnica.com/tech-policy/2025/08/chatgpt-helped-t...

This is the same case that is being discussed, and your comment up-thread does not demonstrate awareness that you are, in fact, agreeing with the parent comment that you replied to. I get the impression that you read only the headline, not the article, and assumed it was a story about someone using ChatGPT for therapy and gaining a positive outcome.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#50

Post by MajimasEyepatch »

sumeno wrote: post #20

This is a clear example of why the people claiming that using a chatbot for therapy is better than no therapy are... I'll be extremely generous and say misguided. This kid wanted his parents to know he was thinking about this and the chatbot talked him out of it.

Exactly right. It's totally plausible that someone could build a mental health chatbot that results in better outcomes than people who receive no support, but that's a hypothesis that can and should be tested and subject to strict ethical oversight.
Post reply on HN