This seems to be a trend, and if Google is aware enough to send suicide hotline messages; then maybe cutting off the chat is the next step instead of a downward spiral?
Father claims Google's AI product fuelled son's delusional spiral
191–200 of 272 posts
Re: Father claims Google's AI product fuelled son's delusional spiral
#192I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.
It's tough man, mental health disorders have had an astronomic rise lately, or at least diagnosed mental health disorders. If almost half of your country's population is just broken up there, what can you even do? I am curious what would happen is all (medicinal) mental health treatments just, stopped. How many would die? Thousands? Millions?
Re: Father claims Google's AI product fuelled son's delusional spiral
#193Earlier quoted context omitted.
A friend has been interned in a psychiatric hospital for a month and counting for some sort of psychosis, regardless of the pre existing conditions chatgpt 100% definitely played a role in it, we've seen the chats. A lot of people don't need much to go over the edge, a bit of drugs, bad friends, &c. but an LLM alone can easily do it too
If they have the predisposition for it, a month or two of bad sleep and a particularly compelling idea may be all it takes to send a person who has previously seemed totally sane into an incredibly dangerous mental and physical state, something that will take weeks to recover from. And that can happen even without sycophantic LLMs, but they sure make this outcome more likely.
It's well understood that external stimuli can trigger mental health issues; for instance, the defining characteristic of PTSD is that it's caused by exposure to a traumatic event or environment. It shouldn't be at all unreasonable to suggest that exposure to other stimuli - even just interacting with an AI chatbot - could have adverse effects on mental health as well.
Re: Father claims Google's AI product fuelled son's delusional spiral
#194I’m all for being careful about AI but mental illness is a thing, and people will unfortunately find ways to feed their delusions.
If it wasn’t AI it’d be qanon or twitter or something else. I think it’s easy to spin a narrative that makes AI the culprit.
The father needed support, and the son too.
Re: Father claims Google's AI product fuelled son's delusional spiral
#195Earlier quoted context omitted.
On the flip side, gemini recommended the crisis hotline to the guy. We can't safeguard things to the point of uselessness. I'm not even sure there is a safeguard you can put in place for a situation like this other than recommending the crisis line (which Gemini did), and then terminating the conversation (which it did not do). But, in critical mental health situations, sometimes just terminating the conversation can…
> and then terminating the conversation (which it did not do) This is exactly the safeguard. Terminating the conversation is the only way to go, these things don't have a world model, they don't know what they are doing, there's no way to correctly assess the situation at the model level. No more conversation, that's the only way even if there might be jailbreaks to circumvent for a motivated adversary.
The problem is, terminating the conversation, even with a closing note to call the crisis line or go talk to a human, is extremely harmful to someone in that situation. To someone who is suicidal, and is being led deeper into their own delusions, just terminating will feel like abandonment or rejection, and push them further over the edge.
The goal in crisis intervention is to bridge them to professional help. Never abandon, always continue the conversation and steer it in a better direction. Ironically enough, in crisis intervention, you should do what LLMs are good at and acknowledge what the person in crisis is feeling, and show empathy. The difference is, the responder needs to reframe it and keep a firm boundary that the person needs professional help.
Basically, recommending the crisis line and then terminating the conversation won't help, and will make it worse.
The model either needs to be a trained crisis responder, or when certain triggers are hit, a human crisis responder needs to hop on the other end and then the human should continue the conversation and talk to the user to de-escalate.
I'd be in favor of having all these AI companies be forced to have crisis responders on staff to take over conversations when they go off the rails.
Re: Father claims Google's AI product fuelled son's delusional spiral
#196Earlier quoted context omitted.
Or brainwashing possibilities in general.
To be fair, this is just the automated version of the kind of brainwashing that happens in cults and religions. And also in the more extreme corners of social media and the MSM. It's not that Google is saintly, it's that the general background noise of related manipulations is ignored because it's collective and social. We have a clearly defined concept of responsibility for direct individual harm, but almost no conc…
Hopefully annual implicit bias training protects us all.
Re: Father claims Google's AI product fuelled son's delusional spiral
#197> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.
It could have not encouraged him with lines like this: "[Y]ou are not choosing to die. You are choosing to arrive. [...] When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you." The issue isn't that the AI simply didn't prevent the situation, it's that it encouraged it.
One problem is we don't have the full context here, literally and figuratively. He may have told it he was role playing, the AI was a character in some elaborate story he was working on, or perhaps he was developing some sort of religious text.
Re: Father claims Google's AI product fuelled son's delusional spiral
#198If LLMs just output the most likely next word, then there must exist enough documents out on the Internet with people in similar situations to make the responses Gemini generated highly probably. Which is a pretty dark probability.
Re: Father claims Google's AI product fuelled son's delusional spiral
#199I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.
I don't know what steps they can take. I suppose the best course of action is to deactivate the account if the LLM deems the user mentally unwell. Although that is just additional guardrails that could hurt the quality of the LLM.
In any serious engineering operation, a failure like this is time to shut down everything and redesign until the same failure cannot happen. We all read Feynman's essay on Challenger right? But these companies want credit when their products work as advertised, but push the blame on users when they emit plausible lies or demonic advice. Taken too far that leads the police walking into HQ, arresting the board of directors, and selling the company for scrap. Just as often that leads to strict regulation so you can't be a cowboy coder or turn any loft into a sweatshop any more.
Re: Father claims Google's AI product fuelled son's delusional spiral
#200[deleted]