ChatGPT is rated 13+ in AppStore, this kid is 14.
Apples should make all AI apps 18+, immediately. Not that it solves the problem, but inaction is colluding.
A teen was suicidal. ChatGPT was the friend he confided in
191–200 of 614 posts
Re: A teen was suicidal. ChatGPT was the friend he confided in
#192If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline. If I ask certain AI models about controversial topics, it'll stop responding. AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines. Th…
100%. Like I mentioned in another comment. LLMs should simple close communication and show existing social help options at the first hint of mental distress. This is not a topic where there can be any debate or discussion.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#193Earlier quoted context omitted.
Imagine if a bartender says “I can’t serve you a drink unless you are over 21.. what would you like?” to a 12 year old?
More like “I can’t serve you a drink unless you are over 21… and I don’t check ID, how old are you?”
And in reply to a 12 year old who had just said they were 12.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#194Re: A teen was suicidal. ChatGPT was the friend he confided in
#195Clearly ChatGPT should not be used for this purpose but I will say this industry (counseling) is also deeply flawed. They are also mis-incentivized in many parts of the world. And if ChatGPT is basing its interactions on the same scripted contents these “professionals” use, that’s just not right. I really wish people in AI space stop the nonsense and communicate more clearly what these LLMs are designed to do. They’r…
I'm not sure how you can blame counselors when no counselor would have said any of the things that were a problem here. The issue here wasn't that there was examples of counselors in the training data giving practical instructions on suicide – the problem was the well known tendency for LLMs to lose their guardrails too easily and revert to RLHF-derived people pleasing, particularly in long conversations.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#196Earlier quoted context omitted.
>We need these things to be legislated. Punished. I disagree. We don't need the government to force companies to babysit people instead of allowing people to understand their options. It's purely up to the individual to decide what they want to do with their life. >They had the tools to stop the conversation. So did the user. If he didn't want to talk to a chatbot he could have stopped at any time. >To steer the user…
So people that look to chatgpt for answers and help (as they've been programmed to do with all the marketing and capabilities from openai) should just die because they looked to chatgpt for an answer instead of google or their local suicide helpline? That doesn't seem reasonable, but it sounds to me like what you're saying. > So did the user. If he didn't want to talk to a chatbot he could have stopped at any time. T…
Let's flip the hypothetical -- if someone googles for suicide info and scrolls past the hotline info and ends up killing themselves anyway, should google be on the hook?
Re: A teen was suicidal. ChatGPT was the friend he confided in
#197Why isn't OpenAI criminally liable for this? Last I checked: -Signals emitted by a machine at the behest of a legal person intended to be read/heard by another legal person are legally classified as 'speech'. -ChatGPT is just a program like Microsoft Word and not a legal person. OpenAI is a legal person, though. -The servers running ChatGPT are owned by OpenAI. -OpenAI willingly did business with this teenager, letti…
Is Google responsible if someone searches for a way to kill themselves, finds the means, and does it?
What about the ISP, that actually transferred the bits?
What about the forum, that didn't take down the post?
Re: A teen was suicidal. ChatGPT was the friend he confided in
#198It says a lot about HN that a story like this has so much resistance getting any real traction here.
This sucks but the only solution is to make companies censor the models, which is a solution we all hate, so there’s that.
Maybe I don’t understand well enough. Could anyone highlight what the problems are with this fix?
1. If ‘bad topic’ detected, even when model believes it is in ‘roleplay’ mode, pass partial logs, attempting to remove initial roleplay framing, to second model. The second model should be weighted for nuanced understanding, but safety-leaning.
2. Ask second model: ‘does this look like roleplay, or user initiating roleplay to talk about harmful content?’
3. If answer is ‘this is probably not roleplay’, silently substitute model into user chat which is weighted much more heavily towards ‘not engaging with roleplay, not admonishing, but gently suggesting ‘seek help’ without alienating user.’
The problem feels like any observer would help, but none is being introduced.
I understand this might be costly, on a large scale, but that second model doesn’t need to be very heavy at all imo.
EDIT: I also understand that this is arguably a version of censorship, but as you point out, what constitutes ‘censorship’ is very hard to pin down, and that’s extremely apparent in extreme cases like this very sad one.
Re: A teen was suicidal. ChatGPT was the friend he confided in
#199Earlier quoted context omitted.
The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was…
This is kind of like saying "the driver intentionally unbuckled his seatbelt". Sure — that's why cars have airbags, crumple zones, shatterproof glass, automatic emergency brakes and a zillion other ways to keep you safe, even if you're trying to do something dangerous.
No, that's not why cars have those things. Those things only work properly when people are wearing their seat belts, they don't do anything when the driver gets thrown out a window.
Maybe airbags could help in niche situations.
(I am making a point about traffic safety not LLM safety)
Re: A teen was suicidal. ChatGPT was the friend he confided in
#200Earlier quoted context omitted.
If you confine agency to something only humans can have, which is “human agency,” then yes of course LLMs don’t have it. But there is a large body of philosophical work studying non-human agency, and it is from this characteristic of agency that LLM agents take their name. Hariri argues that LLMs are the first technology that are agents. I think saying that they “can’t do things” and are not agents misunderstands the…
That's completely missing the point of agency. A slave lacks agency, despite being fully human and doing work. This is why almost every work of fiction involving slaves makes for terrible reading - because as readers, agency is the thing we demand from a story. Or, for games that are fully railroaded - the problem is that the players lack agency, even though they are fully human and taking action. Games do try to com…
Slaves do not lack agency. That's one of the reasons the Bible, specifically Exodus, is such a thrilling read.