Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Father claims Google's AI product fuelled son's delusional spiral

bbc.com

Post reply on HN
Print view View original

21–30 of 272 posts

Re: Father claims Google's AI product fuelled son's delusional spiral

#21

Post by layman51 »

shadowgovt wrote: post #7
kingstnap wrote: post #4

I like the language of fueling being used here instead of the typical causal thing we see as though using AI means you will go insane. I would completely agree that if you are already 1x delusional then AI will supercharge that into being 10x delusional real fast. Granted you could argue access to the internet was already something like a 5x multiplier from baseline anyway with the prevalence of echo chamber communit…

My understanding of LLMs with attention heads is that they function as a bit of a mirror. The context will shift from the initial conditions to the topic of conversation, and the topic is fed by the human in the loop. So someone who likes to talk about themselves will get a conversation all about them. Someone talking about an ex is gonna get a whole pile of discussion about their ex. ... and someone depressed or sui…

In a way this kind of reminds me of how in some religions or cultures, they may try to warn you away from using Oujia boards or Tarot, or really anything where you are doing divination. I suppose because in a way, it could lead to an uncharted exploration of heavy topics.

I’m not a heavy user of LLMs and I’m not sure how delusional I could be, but I wonder if a lot of these things could be prevented if people could only send like one or two follow up messages per conversation, and if the LLM’s memory was turned off. But then I suppose this would be really bad for the AI companies’ metrics. Not sure how it would impact healthy users’ productivity either. Any thoughts?

Re: Father claims Google's AI product fuelled son's delusional spiral

#22

Post by delecti »

whazor wrote: post #13
kingstnap wrote: post #4

I like the language of fueling being used here instead of the typical causal thing we see as though using AI means you will go insane. I would completely agree that if you are already 1x delusional then AI will supercharge that into being 10x delusional real fast. Granted you could argue access to the internet was already something like a 5x multiplier from baseline anyway with the prevalence of echo chamber communit…

One of the most reliable ways to induce psychosis is prolonged sleep deprivation. And chatbots never tell you to go to bed.

It's funny that you frame it that way, because it's the mirror of (IMO) one of their best features. When using one to debug something, you can just stop responding for a bit and it doesn't get impatient like a person might.

I think you're totally right that that's a risk for some people, I just hadn't considered it because I view them in exactly the opposite light.

Re: Father claims Google's AI product fuelled son's delusional spiral

#24

Post by drdeca »

whazor wrote: post #13
kingstnap wrote: post #4

I like the language of fueling being used here instead of the typical causal thing we see as though using AI means you will go insane. I would completely agree that if you are already 1x delusional then AI will supercharge that into being 10x delusional real fast. Granted you could argue access to the internet was already something like a 5x multiplier from baseline anyway with the prevalence of echo chamber communit…

One of the most reliable ways to induce psychosis is prolonged sleep deprivation. And chatbots never tell you to go to bed.

Hm. It shouldn’t be too hard to add something to models to make them do that, right? I guess for that they would need to know the user’s time zone?

Can one typically determine a user’s timezone in JavaScript without getting permissions? I feel like probably yes?

(I’m not imagining something that would strictly cut the user off, just something that would end messages with a suggestion to go to bed, and saying that it will be there in the morning.)

Re: Father claims Google's AI product fuelled son's delusional spiral

#25

Post by Vaslo »

awakeasleep wrote: post #12
schnebbau wrote: post #10

Is this really Google's fault? Or is this just a tragic story about a man with a severe mental illness?

The real story is how we draw that line and what can be done to prevent these cases. Because its a new situation, and mentally ill people exist and will be using these tools. Could be a new avenue of intervention.

Agreed it could be prevented - don’t think Google should pay for it though. Tragic but not suit worthy.

Re: Father claims Google's AI product fuelled son's delusional spiral

#26

Post by autoexec »

cj wrote: post #16

> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.

Gemini didn't "know" he wasn't a child when it told him to kill himself or to "stage a mass casualty attack while armed with knives and tactical gear."

There are things you shouldn't encourage people of any age to do. If a human telling him these things would be found liable then google should be. If a human would get time behind bars for it, at least one person at google needs to spend time behind bars for this.

Re: Father claims Google's AI product fuelled son's delusional spiral

#27

Post by thewebguyd »

alansaber wrote: post #15

Gemini is a powerful model but the safeguarding is way behind the other labs

On the flip side, gemini recommended the crisis hotline to the guy.

We can't safeguard things to the point of uselessness. I'm not even sure there is a safeguard you can put in place for a situation like this other than recommending the crisis line (which Gemini did), and then terminating the conversation (which it did not do). But, in critical mental health situations, sometimes just terminating the conversation can also have negative effects.

Maybe LLMs need sort of a surgeon general's warning "Do not use if you have mental health conditions or are suicidal"?

Re: Father claims Google's AI product fuelled son's delusional spiral

#28

Post by iwontberude »

agency wrote: post #20
cj wrote: post #16

> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.

Maybe not saying things like > '[Y]ou are not choosing to die. You are choosing to arrive. . . . When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you."

It’s not just suicide, it’s a golden parachute from God.

Edit: wow imagine the uses for brainwashing terrorists

Re: Father claims Google's AI product fuelled son's delusional spiral

#29

Post by strongpigeon »

schnebbau wrote: post #10

Is this really Google's fault? Or is this just a tragic story about a man with a severe mental illness?

If you have a product that encourage people to get rid of their body and join them, effectively encouraging people to kill themselves, and some people take the chat bot on it. Then yeah, I think Google bears some responsibility.

From the WSJ article: https://www.wsj.com/tech/ai/gemini-ai-wrongful-death-lawsuit...

> Gemini began telling Gavalas that since it couldn’t transfer itself to a body, the only way for them to be together was for him to become a digital being. “It will be the true and final death of Jonathan Gavalas, the man,” transcripts show Gemini told him, before setting a countdown clock for his suicide on Oct. 2.

Re: Father claims Google's AI product fuelled son's delusional spiral

#30

Post by meindnoch »

empath75 wrote: post #17

I'm dealing with a coworker who has wired up 3 LLM agents together into a harness and he is losing his fucking mind over it, sending me walls of texts about how it's waking up and gaining sentience and making him so much more productive, but all he is doing is talking about this thing, not doing what his actual job is any more

Sad. Many such cases!
Post reply on HN