A teen was suicidal. ChatGPT was the friend he confided in • Hacker News Board

Re: A teen was suicidal. ChatGPT was the friend he confided in

#481

Post by kunley » Wed, Aug 27, 2025, 4:08 PM UTC

jackjeff wrote: post #188

slg wrote: post #79

Earlier quoted context omitted.

Thank you, “we just have to accept that these systems will occasionally kill children” is a perfect example of the type of mindset I was criticizing.

Don’t cars and ropes and drills occasionally kill people too? Society seems to have accepted that fact long ago. Somehow we expect the digital world to be devoid of risks. Cryptography that only the good guys can crack is another example of this mindset. Now I’m not saying ClosedAI look good on this, their safety layer clearly failed and the sycophantic BS did not help. But I reckon this kind of failure more will alw…

Cars and ropes don't "talk" and don't make an impression of a human being.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#482

Post by kunley » Wed, Aug 27, 2025, 4:09 PM UTC

ares623 wrote: post #369

rsynnott wrote: post #311

Earlier quoted context omitted.

"It is difficult to get a man to understand something, when his salary depends on his not understanding it."

Ironically, the salary of a majority of us here actually depends on AI getting nipped in the bud.

Ditto

Re: A teen was suicidal. ChatGPT was the friend he confided in

#483

Post by kunley » Wed, Aug 27, 2025, 4:13 PM UTC

kouteiheika wrote: post #278

slg wrote: post #79

Earlier quoted context omitted.

Thank you, “we just have to accept that these systems will occasionally kill children” is a perfect example of the type of mindset I was criticizing.

So what's the alternative? Pervasive censorship and horrible privacy and freedom-of-speech destroying legislation like UK's Online Safety Act? I'm not looking forward to the day when half of the Internet will require me to upload my ID to verify that I'm an adult, and the other half will be illegal/blocked because they refuse to do the verification. But yeah, think of the children!

The alternative is as usual, getting certain individuals to the court and making them pay for the damage or even to go to jail, instead of preemptively trying to give them a holy cow status.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#484

Post by novok » Wed, Aug 27, 2025, 4:13 PM UTC

idle_zealot wrote: post #456

novok wrote: post #220

Earlier quoted context omitted.

After a certain point, people are responsible for what they do when they see certain words, especially words they know to be potentially inaccurate, fictional and have a lot of time to consider the actual reality of. A book is not responsible for people doing bad things, they are themselves. AI models are similar IMO, and unlike fiction books are often clearly labeled as such, repeatedly. At this point if you don't k…

> At this point if you don't know if an AI model is inaccurate and do something seriously bad, you should probably be a ward of the state. You either think too highly of people, or too lowly of them. In any case, you're advocating for interning about 100 million individuals.

It was a joke

Re: A teen was suicidal. ChatGPT was the friend he confided in

#485

Post by novok » Wed, Aug 27, 2025, 4:15 PM UTC

latexr wrote: post #236

novok wrote: post #227

Earlier quoted context omitted.

You do have empathy for the person who had a tragedy, but it doesn't mean you go into full safetyism / scapegoating that causes significantly less safety and far more harm because of the emotional weight of something in the moment. It's like making therapists liable for people committing suicide or for people with eating disorders committing suicide indirectly. What ends up happening when you do is therapists avoidin…

That is one big slippery slope fallacy. You are inventing motives, outcomes, and future unproven capabilities out of thin air. It’s a made up narrative which does not reflect the state of the world and requires one to buy into a narrow, specific world view. https://en.wikipedia.org/wiki/Slippery_slope

Instead of just saying “thats not true”, could you actually point out how it is not?

Re: A teen was suicidal. ChatGPT was the friend he confided in

#486

Post by kunley » Wed, Aug 27, 2025, 4:17 PM UTC

lvl155 wrote: post #39

Clearly ChatGPT should not be used for this purpose but I will say this industry (counseling) is also deeply flawed. They are also mis-incentivized in many parts of the world. And if ChatGPT is basing its interactions on the same scripted contents these “professionals” use, that’s just not right. I really wish people in AI space stop the nonsense and communicate more clearly what these LLMs are designed to do. They’r…

Gen pop doesn't have a clue about what they're getting at so many levels of life. The communication effort you're mentioning is needed even more then

Re: A teen was suicidal. ChatGPT was the friend he confided in

#487

Post by nozzlegear » Wed, Aug 27, 2025, 4:20 PM UTC

scotty79 wrote: post #414

franktankbank wrote: post #406

Earlier quoted context omitted.

This is one model though. "I'm sorry I'm censored but if you like I can cosplay quite effectively as an uncensored one." So you're not censored really?

Societies love theatres. Model guardrails are for chats what TSA is for air travel.

Society loves teenagers not being talked into suicide by a billionaire's brainchild. That's not theater.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#488

Post by AnIrishDuck » Wed, Aug 27, 2025, 4:21 PM UTC

bastawhiz wrote: post #474

AnIrishDuck wrote: post #168

Earlier quoted context omitted.

> ChatGPT is a program. The kid basically instructed it to behave like that. I don't think that's the right paradigm here. These models are hyper agreeable. They are intentionally designed to mimic human thought and social connection. With that kind of machine, "Suicidal person deliberately bypassed safeguards to indulge more deeply in their ideation" still seems like a pretty bad failure mode to me. > Vanilla OpenAI…

> These models are hyper agreeable. They are intentionally designed to mimic human thought and social connection. Python is hyper agreeable. If I comment out some safeguards, it'll happily bypass whatever protections are in place. Lots of people on here argue vehemently against anthropomorphizing LLMs. It's either a computer program crunching numbers, or it's a nebulous form of pseudo-consciousness, but you can't hav…

> Python is hyper agreeable. If I comment out some safeguards, it'll happily bypass whatever protections are in place.

These models are different from programming languages in what I consider to be pretty obvious ways. People aren't spontaneously using python for therapy.

> Lots of people on here argue vehemently against anthropomorphizing LLMs.

I tend to agree with these arguments.

> It's either a computer program crunching numbers, or it's a nebulous form of pseudo-consciousness, but you can't have it both ways. It's either a tool that has no mind of its own that follows instructions, or it thinks for itself.

I don't think that this follows. I'm not sure that there's a binary classification between these two things that has a hard boundary. I don't agree with the assertion here that these things are a priori mutually exclusive.

> I'm not arguing that the model behaved in a way that's ideal, but at what point do you make the guardrails impassable for 100% of users? How much user intent do you reject in the interest of the personal welfare of someone intent on harming themselves?

These are very good questions that need to be asked when modifying these guardrails. That's all I'm really advocating for here: we probably need to rethink them, because they seem to have major issues that are implicated in some pretty terrible outcomes.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#489

Post by teaearlgraycold » Wed, Aug 27, 2025, 4:27 PM UTC

blackqueeriroh wrote: post #247

teaearlgraycold wrote: post #161

Earlier quoted context omitted.

I really want the LLMs to respond like a senior developer that doesn't have time for you but needs you to get your job done right. A little rude and judgemental, but also highly concise.

You say that now, but how they actually behave says that you’d probably get tired of it.

I’m not the one providing the RLHF feedback. They’re optimized for the lowest common denominator.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#490

Post by nozzlegear » Wed, Aug 27, 2025, 4:30 PM UTC

kouteiheika wrote: post #376

KaiserPro wrote: post #285

Earlier quoted context omitted.

I hate to be all umacksually about this, but a flaw is still a tradeoff. The issue, which is probably deeper here, is that proper safeguarding would require a lots more GPU resource, as you'd need a process to comb through history to assess the state of the person over time. even then its not a given that it would be reliable. However it'll never be attempted because its too expensive and would hurt growth.

> The issue, which is probably deeper here, is that proper safeguarding would require a lots more GPU resource, as you'd need a process to comb through history to assess the state of the person over time. > > even then its not a given that it would be reliable. However it'll never be attempted because its too expensive and would hurt growth. There's no "proper safeguarding". This isn't just possible with what we have…

> Again, this is a tradeoff. It's not a binary issue of "doing it properly". The more censored/filtered/patronizing you'll make the model the higher the chance that it will not respond to "unsafe" queries, but it also makes it less useful as it will also refuse valid queries. [..] So where do we draw the line?

That sounds like a tough problem for OpenAI to figure out. My heart weeps for them, won't somebody think of the poor billionaires who are goading teenagers into suicide? Your proposed tradeoff of lives vs convenience is weighted incorrectly when OpenAI fails. Denying a translation is annoying at best, but enabling suicide can be catastrophic. The convenience is not morally equal to human life.

> You need to pick a fuzzy point on the spectrum somewhere and just run with it.

My fuzzy point is not fuzzy at all: don't tell people how to kill themselves, don't say "I can't help you with that but I could roleplay with you instead". Anything less than that is a moral failure on Sam Altman and OpenAI's part, regardless of how black the box is for their engineers.