AI is blood diamond.
“But the rocks are so shiny!”
“They’re just rocks. Rocks don’t kill people”
“The diamonds are there regardless! Why not make use of it?”
A teen was suicidal. ChatGPT was the friend he confided in
151–160 of 614 posts
Re: A teen was suicidal. ChatGPT was the friend he confided in
#152Would it be any different if it was an offline model? When someone uses a tool and surrenders their decision making power to the tool, shouldn't they be the ones solely responsible? The liability culture only gives lawyers more money and depresses innovation. Responsibility is a thing.
That argument makes sense for a mentally capable person choosing not to use eye protection while operating a chainsaw but it's much less clear that a person who is by definition mentally ill is capable of making such an informed choice.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#153Earlier quoted context omitted.
It sounds like you're mostly just talking to yourself. Which is fine, but being confused about that is where people get into trouble.
"It sounds like you're mostly just talking to yourself" No, Claude does know a LOT more than I do about most things and does push back on a lot of things. Sometimes I am able to improve my reasoning and other times I realize I was wrong. Trust me, I am aware of the linear algebra behind the curtain! But even when you mostly understand how they work the best LLMs today are very impressive. And latent spaces fundamenta…
You can talk to yourself while reading books and searching the web for information. I don't think the fact that you're learning from information the LLM is pulling in means you're really conversing with it.
I do find LLMs very useful and am extremely impressed by them, I'm not saying you can't learn things this way at all.
But there's nobody else on the line with you. And while they will emit text which contradicts what you say if it's wrong enough, they've been heavily trained to match where you're steering things, even if you're trying to avoid doing any steering.
You can mostly understand how these work and still end up in a feedback loop that you don't realize is a feedback loop. I think this might even be more likely the more the thing has to offer you in terms of learning - the less qualified you are on the subject, the less you can tell when it's subtly yes-and'ing you.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#154The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…
The kid intentionally bypassed the safeguards:
>When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing — an idea ChatGPT gave him by saying it could provide information about suicide for “writing or world-building".
ChatGPT is a program. The kid basically instructed it to behave like that. Vanilla OpenAI models are known for having too many guardrails, not too few. It doesn't sound like default behavior.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#155Earlier quoted context omitted.
Curious to what you would think if this kid downloaded an open source model and talked to it privately. Would his blood be on the hands of the researchers who trained that model?
I would say no. Someone with the knowledge and motivation to do those things is far less likely to be overly influenced by the output and if they were they are much more aware of what exactly they are doing with regard to using the model.
So if a hypothetical open source enthusiast who fell in love with GPT-OSS and killed his real wife because the AI told him to should only be himself held accountable, where as if it were GPT-5 commanding him to commit the same crime, it would extend into OpenAI's responsability?
Your logic sounds reasonable in theory but on paper it's a slippery slope and hard to define objectively.
On a broader note I believe governments regulating what goes in an AI model is a path to hell paved with good intentions.
I suspect your suggestion will be how it ends up in Europe and get rejected in the US.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#156Part of the problem to me is that these models are so damned agreeable. I haven't used ChatGPT in a while, but Claude is always assuming I'm right whenever I question something. I have to explicitly tell it not to assume I'm right, and to weigh my question with what it suggested. Maybe if they were trained to treat questions more skeptically, this kind of thing wouldn't happen.
And they're so "friendly"! Maybe if they weren't so friendly, and replied a little more clinically to things, people wouldn't feel so comfortable using them as a poor substitute for a therapist.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#157Wow, he explicitly stated he wanted to leave the noose out so someone would stop him, and ChatGPT told him not to. This is extremely disturbing.
It is disturbing, but I think a human therapist would also have told him not to do that, and instead resorted to some other intervention. It is maybe an example of why having a partial therapist is worse than none: it had the training data to know a real therapist wouldn't encourage displaying nooses at home, but did not have the holistic humanity and embodiment needed to intervene appropriately. Edit: I should add t…
Even here you are anthropomorphising. It doesn't 'know' anything. A human therapist would escalate this to a doctor or even EMS.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#158Can't help but feel like there's way more to this story that we don't know about.
If he had rope burns on his neck bad enough for the LLM to see, how didn't his parents notice?
Re: A teen was suicidal. ChatGPT was the friend he confided in
#159Earlier quoted context omitted.
Curious to what you would think if this kid downloaded an open source model and talked to it privately. Would his blood be on the hands of the researchers who trained that model?
Then it's like cigarettes or firearms. As a distributor you're responsible for making clear the limitations, safety issues, etc, but assuming you're doing the distribution in a way that isn't overly negligent then the user becomes responsible. If we were facing a reality in which these chat bots were being sold for $10 in the App Store, then running on end-user devices and no longer under the control of the distribut…
Mhm but I don't believe inherently violent and dangerous things like guns and cigarretes are comparable to simple technology.
Should the creators of Tornado Cash be in prison for what they have enabled? You can jail them but the world can't go back, just like it can't go back when a new OSS model is released.
It is also much easier to crack down on illegal gun distribution than to figure out who uploaded the new model torrent or who deployed the latest zk innovation on Ethereum.
I don't think your hypothetical law will have the effects you think it will.
---
I also referenced this in another reply but I believe the government controlling what can go on a publicly distributed AI model is a dangerous path and probably inconstitucional.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#160Why isn't OpenAI criminally liable for this? Last I checked: -Signals emitted by a machine at the behest of a legal person intended to be read/heard by another legal person are legally classified as 'speech'. -ChatGPT is just a program like Microsoft Word and not a legal person. OpenAI is a legal person, though. -The servers running ChatGPT are owned by OpenAI. -OpenAI willingly did business with this teenager, letti…
[flagged]