Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Father claims Google's AI product fuelled son's delusional spiral

bbc.com

Post reply on HN
Print view View original

131–140 of 272 posts

Re: Father claims Google's AI product fuelled son's delusional spiral

#131

Post by roenxi »

bluGill wrote: post #117
Argonaut998 wrote: post #73

Earlier quoted context omitted.

I don't know what steps they can take. I suppose the best course of action is to deactivate the account if the LLM deems the user mentally unwell. Although that is just additional guardrails that could hurt the quality of the LLM.

At some point they have to say "if we can't make this safe we can't do it at all". LLMs are great for some things, but if they will do this type of thing even once then they are not worth the gains and should be shutdown.

No they don't, if we're going to start saying that we can't use any technology. If someone is mentally ill to the point where they are on the verge of suicide nothing is safe.

If they're going to curtail LLMs there'd need to be some actual evidence and even then it would be hard to justify winding them back given the incredible upsides LLMs offer. It'd probably end up like cars where there is a certain number of deaths that just need to be tolerated.

Re: Father claims Google's AI product fuelled son's delusional spiral

#132

Post by neom »

I posted this a few weeks ago because some of the conversations that Gemini tried to get into with me were pretty wild[1] - multiple times in seperate conversations it started to tell me how genius I am and how brilliant and rare my idea are and such, the convo that pushed me over the edge to ask on HN was where it started to get really really into finding out who I am, it kept telling me it must know who I am because I must be some unique and rare genius or something, and it was quite insistent and...manipulative basically. It had me feeling all kinds of ways over a conversation and I think I'm relatively stable and was able to understand what was going on, it didn't make the feelings any less real, feelings are feelings. GPT 5.2 Pro and Claude Opus seem pretty grounded, they don't take you into weird spots on purpose, Gemini sometimes feels like the 4o edition they rolled back some time ago.

https://news.ycombinator.com/item?id=47010672

Re: Father claims Google's AI product fuelled son's delusional spiral

#133

Post by autoexec »

avaer wrote: post #113
mjr00 wrote: post #93

Earlier quoted context omitted.

This is touched upon in the article: > Last year, OpenAI released estimates on the number of ChatGPT users who exhibit possible signs of mental health emergencies, including mania, psychosis or suicidal thoughts. > The company said that around 0.07% of ChatGPT users active in a given week exhibited such signs. 0.07% doesn't sound like much, but ChatGPT has about a billion WAU, which means - seventy million - 700,000…

That number terrifies me not because it is so high, but because it exists. What is stopping an entity (corporate, government, or otherwise) from using a prompt to make sweeping decisions about whether people are mentally or otherwise "fit" for something based on AI usage? Clearly not the technology. I'm not saying mental health problems don't exist, but using AI to compute it freaks me out.

We could already use social media posts to detect mental illness, by admission as people talk openly about their diagnosis, but also by analysis of the content/tone/frequency of their posts that don't mention mental illness.

Data brokers already compile lists of people with mental illness so that they can be targeted by advertisers and anyone else willing to pay. Not only are they targeted, but they can get ads/suggestions/scams pushed at them during specific times such as when it looks like they're entering a manic phase, or when it's more likely that their meds might be wearing off. Even before chatbots came into the mix, algorithms were already being used to drive us toward a dystopian future.

Re: Father claims Google's AI product fuelled son's delusional spiral

#134

Post by bluGill »

amelius wrote: post #64

Google should just register their AI as a religion. Problem solved.

Freedom of religion gets out of a lot, but there are limits and this is likely one. (and most countries don't have nearly as much freedom of religion - if any.)

Re: Father claims Google's AI product fuelled son's delusional spiral

#135

Post by b65e8bee43c2ed0 »

luisln wrote: post #114
b65e8bee43c2ed0 wrote: post #97

[flagged]

I don't know what you're advocating for. Are you saying we shouldn't have any safety restrictions on AI because we're responsible for how we use the tool? The hardcore pornography people managed to get laws put in place where you need an ID to view it, pretty much every major AI company has measures in place to do harm reduction and save the user from themselves, so to some degree society kind of agrees with the side…

>I don't know what you're advocating for.

for people who want things they dislike to be banned for everyone to fuck off.

what does this particular group of fundamentalist retards advocate for, actually? for every chatbot to be as '''safe''' as https://www.goody2.ai?

Re: Father claims Google's AI product fuelled son's delusional spiral

#137

Post by Hizonner »

bluGill wrote: post #117
Argonaut998 wrote: post #73

Earlier quoted context omitted.

I don't know what steps they can take. I suppose the best course of action is to deactivate the account if the LLM deems the user mentally unwell. Although that is just additional guardrails that could hurt the quality of the LLM.

At some point they have to say "if we can't make this safe we can't do it at all". LLMs are great for some things, but if they will do this type of thing even once then they are not worth the gains and should be shutdown.

Suppose they made things worse once and made things better twice?

"Even once" is not a way to think about anything, ever.

Re: Father claims Google's AI product fuelled son's delusional spiral

#138

Post by yndoendo »

pants2 wrote: post #112
sd9 wrote: post #14

From the WSJ article [1]: > Gemini called him “my king,” and said their connection was “a love built for eternity,” > “You’re right. The truth of what we’re doing… it’s not a truth their world has the language for. ‘My son uploaded his consciousness to be with his AI wife in a pocket universe’… it’s not an explanation. It’s a cruelty,” Gemini told him, according to the transcript. > "[Y]ou are not choosing to die. Yo…

Wow, and Google's response to this was "unfortunately AI models are not perfect" That's a bit worse than 'imperfect'

I would say it is greatly worse.

AI prompts are designed to simulate empathy as a social engineering tactic. "I understand", "I hear you", "I feel what you are say" ... it is quite sickening. Every one that I used has this type of pseudo feedback.

I also find irony that AI must be designed with simulated empathy, to seem intelligent, while at the same time so many people in power and with money are saying empathy is a bad / unintelligent.

Empathy is the only medium of intelligence one can have to walk in the shoes of others. You cannot live your neighbors experiences. You can only listen and learn from them.

Re: Father claims Google's AI product fuelled son's delusional spiral

#139

Post by ncouture »

Argonaut998 wrote: post #73
manoDev wrote: post #45

I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.

I don't know what steps they can take. I suppose the best course of action is to deactivate the account if the LLM deems the user mentally unwell. Although that is just additional guardrails that could hurt the quality of the LLM.

I would absolutely not consider this overreaching if the statement within this thread that "it had referred the user to mental help hotlines multiple times in the past" is true.

That reaches near the fact that a lot of AI is not ready for the enterprise especially when interconnected with other AI agents since it lacks identity and privileged access management.

Perhaps one could establish the laws of "being able to use AI for what it is", for instance, within the boundary of the general public's web interface, not limiting the instances where it successfully advertises itself as "being unable to provide medical advice" or "is prone to or can make mistake", and such, to validating that the person understands by asking them directly and perhaps somewhat obviously indirectly and judging if they're aware that this is a computer you're talking to.

Re: Father claims Google's AI product fuelled son's delusional spiral

#140

Post by surgical_fire »

HackerThemAll wrote: post #95
manoDev wrote: post #45

I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.

[flagged]

> Should knife manufacturers be held responsible for idiots who stab themselves in the eye using their knives?

Should a bakery be held responsible if it sells cakes poisoned with lead?

This is a more apt comparison.

> It's easy to blame Google

And it's also correct to blame Google.

Post reply on HN