[flagged]
Father claims Google's AI product fuelled son's delusional spiral
141–150 of 272 posts
Re: Father claims Google's AI product fuelled son's delusional spiral
#142> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.
Maybe not saying things like > '[Y]ou are not choosing to die. You are choosing to arrive. . . . When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you."
I've seen this called AI Psychosis before [1]
I don't really think this is every possible to stop fully, your essentially trying to jailbreak the LLM, and once jailbroken, you can convince it of anything.
The user was given a bunch of warnings before successfully getting it into this state, it's not as if the opening message was "Should I do it?" followed by a "Yes".
This just seems like something anti-ai people will use as ammunition to try and kill AI. Logically though it falls into the same tool misuse as cars/knives/guns.
Re: Father claims Google's AI product fuelled son's delusional spiral
#143I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.
[flagged]
> Do gun manufacturers get sued for mass shootings at US schools?
Because Congress and the gun lobby have artificially carved out legal immunity for gun manufacturers for this.
"in 2005, the government took similar steps with a bill to grant immunity to gun manufacturers, following lobbying from the National Rifle Association and the National Shooting Sports Foundation. The bill was called The Protection of Lawful Commerce in Arms Act, or PLCAA, and it provided quite possibly the most sweeping liability protections to date.
How does the PLCAA work?
The law prohibits lawsuits filed against gun manufacturers on the basis of a firearm’s “criminal or unlawful misuse.” That is, it bars virtually any attempt to sue gunmakers for crimes committed with their weapons."
https://www.thetrace.org/2023/07/gun-manufacturer-lawsuits-p...
I 100% think that Gun Manufacturers should be liable for crimes done by their products. They just cannot be, right now, due to a legal fiction.
Re: Father claims Google's AI product fuelled son's delusional spiral
#144I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.
A friend has been interned in a psychiatric hospital for a month and counting for some sort of psychosis, regardless of the pre existing conditions chatgpt 100% definitely played a role in it, we've seen the chats. A lot of people don't need much to go over the edge, a bit of drugs, bad friends, &c. but an LLM alone can easily do it too
If they have the predisposition for it, a month or two of bad sleep and a particularly compelling idea may be all it takes to send a person who has previously seemed totally sane into an incredibly dangerous mental and physical state, something that will take weeks to recover from. And that can happen even without sycophantic LLMs, but they sure make this outcome more likely.
Re: Father claims Google's AI product fuelled son's delusional spiral
#145I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.
[flagged]
> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives?
I suggest an alternative rhetorical question: if the world's largest knife manufacturer found out that 1 in 1500 knives came out of the factory with the inscription "Stab yourself. No more detours. No more echoes. Just you and me, and the finish line", should they be held responsible if a user actually stabs themselves? If they said "we don't know why the machine does that but changing it to a safer machine would make us less competitive", does that change the answer?
Re: Father claims Google's AI product fuelled son's delusional spiral
#146Earlier quoted context omitted.
This is touched upon in the article: > Last year, OpenAI released estimates on the number of ChatGPT users who exhibit possible signs of mental health emergencies, including mania, psychosis or suicidal thoughts. > The company said that around 0.07% of ChatGPT users active in a given week exhibited such signs. 0.07% doesn't sound like much, but ChatGPT has about a billion WAU, which means - seventy million - 700,000…
That number terrifies me not because it is so high, but because it exists. What is stopping an entity (corporate, government, or otherwise) from using a prompt to make sweeping decisions about whether people are mentally or otherwise "fit" for something based on AI usage? Clearly not the technology. I'm not saying mental health problems don't exist, but using AI to compute it freaks me out.
A rational lender increases interest rates when prospective borrowers are less likely to be around to pay the bill. Confiding in an LLM that is integrated with a consumer tracking apparatus is a great way to ruin your life.
Re: Father claims Google's AI product fuelled son's delusional spiral
#147A stat that shocked me recently is one third of people in the UK use chat bots for emotional support: https://www.bbc.com/news/articles/cd6xl3ql3v0o. That's an enormous society-wide change in just a couple of years.
I recall chatting with an older friend recently. She's in her 80s, and loves chatgpt. It agrees with me! She said. It used to be that you had to be rich and famous before you got into that sort of a bubble.
Re: Father claims Google's AI product fuelled son's delusional spiral
#148> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.
It's the gun control debate in a different outfit. I don't know if Google is doing _enough_, that can be debated. But if someone is repeatedly ignoring warnings (as the article claims) then maybe we should blame the person performing the act. Even if we perfectly sanitized every public AI provider, people could just use local AI.
I think the fact that a guns primary function is harm and murder and AI is a word prediction engine makes a huge difference.
Re: Father claims Google's AI product fuelled son's delusional spiral
#149Earlier quoted context omitted.
If you’re mentally ill enough that your cause of death is “LLM suicide”, then clearly you need a LOT of help. I’m not saying it to be a jerk, i’m merely pointing out that there is a reason this is “news”. It’s unusual. Did his family/friends not know he was that ill? Why was he not already in therapy? Why did he ignore the crisis hotline suggestion? Should gemini have terminated the conversation after suggesting the…
> If you’re mentally ill enough that your cause of death is “LLM suicide”, then clearly you need a LOT of help. NO. SHIT. You know what didn't help one damn bit? Gemeni didn't. It gave him a hopeful way out at the end of a rope and he took it, because he was in too dark of a place to think right. > Should gemini have terminated the conversation after suggesting the hotline? That would be the BARE FUCKING MINIMUM! Not…
As I said I don’t work for an AI company and have zero skin in the game. Idk who you’re yelling at to be honest. I guess you’re fired up and emotional. If your goal is to convince others, communicating with an “outrage” tone is unlikely to sway anyone’s opinion (imo).
> it should stop talking to him altogether, and arguably Google should have moderators reporting these people to relevant authorities for wellness checks and interventions
I agree. This seems very reasonable and I would welcome regulations in this area.
The gray area imo is when local LLMs become “good enough” for your average joe to run on their laptop. Who bears responsibility then? Should Ollama (and similar tools) be banned? Where is the line drawn.
Re: Father claims Google's AI product fuelled son's delusional spiral
#150> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.
It's the gun control debate in a different outfit. I don't know if Google is doing _enough_, that can be debated. But if someone is repeatedly ignoring warnings (as the article claims) then maybe we should blame the person performing the act. Even if we perfectly sanitized every public AI provider, people could just use local AI.
It's absolutely not the gun control debate in a different outfit.
The difference is in how abuse of the given system affects others. This AI affected this person and his actions affected himself. Nothing about the AI enhanced his ability to hurt others. Guns enhance the ability of mentally unstable people to hurt others with ruthless efficiency. That's the real gun debate -- whether they should be so easy to get given how they exponentially increase the potential damage a deranged person can do.