Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

People are losing loved ones to AI-fueled spiritual fantasies

rollingstone.com

Post reply on HN
Print view View original

81–90 of 173 posts

Re: People are losing loved ones to AI-fueled spiritual fantasies

#81

Post by bell-cot »

stevage wrote: post #65
bell-cot wrote: post #56

Earlier quoted context omitted.

If you were hit and badly injured by brand-new model of car, where would you want the ambulance to take you? - the dealership that sold that car, where they know all about it - a hospital emergency room, where they have a lot of experience with patients injured by other, different models of car I'm thinking that the age-old commonality on the human side matters far more than the transient details on the obsession/add…

Your comment above reads more like, let's not even discuss the fact that new models of cars are killing pedestrians in greater numbers than before, since pedestrians have always been killed by cars.

Re-skimming the article, I failed to spot the fact that this AI stuff is claiming more victims than earlier flavors of rabbit hole did. Was that in content which the article linked to?

Because if the new model cars aren't statistically more dangerous to pedestrians, then public safety efforts should be focused on things like getting the pedestrians to look up from their phones when crossing the street. Not "OMG! New 2025-model cars can hurt pedestrians who wander in front of them!" panics.

(Note that I'm old enough to remember when people were going down the rabbit hole of angry conspiracy theories spread via email. And when typical download speeds got high enough to make internet porn video addictions workable. And when loved ones started being lost to "EverCrack" ( https://en.wikipedia.org/wiki/EverQuest ). And when ...)

Re: People are losing loved ones to AI-fueled spiritual fantasies

#82

Post by nico »

kaycey2022 wrote: post #60

Looks like Chatgpt persists some context information across chats and doesn't ever delete these profiles. Worst case would be for this to persist across users. That isn't unlikely given the stories of them leaking API keys etc.

That’s essentially what Google, Facebook, banks, financial institutions and even retail, have been doing for a long time now

People’s data rarely gets actually deleted. And it gets actively sold as well as used to track and influence us

Can’t say for the specifics of what ChatGPT is or will be doing, but imagine what Google already knows about us just with their maps app, search, chrome and Android phones

Re: People are losing loved ones to AI-fueled spiritual fantasies

#85

Post by heavyset_go »

zdragnar wrote: post #40
lr4444lr wrote: post #19

[flagged]

It'd be great if it were trained on therapeutic resources, but otherwise just ends up enabling and amplifying the problem. I knew of someone who had paranoid delusions and schizophrenia. He didn't like taking his medicine due to the side effects, but became increasingly convinced that vampires were out to kill him. Friends, family and social workers could help him get through episodes and back on the medicine before…

> I'm terrified that people like him will push away friends and family because the LLM engages with their delusions.

There's that danger from the internet, as well as the danger of being exposed to conmen that are okay with exploiting mental illness for profit. Watched this happen to an old friend with schizophrenia.

There are online communities that are happy to affirm delusions and manipulate sick people for some easy cash. LLMs will only make their fraud schemes more efficient, as well.

Re: People are losing loved ones to AI-fueled spiritual fantasies

#86

Post by heavyset_go »

patrickhogan1 wrote: post #66
Barrin92 wrote: post #61

Earlier quoted context omitted.

>’90s and early ’00s about kids who couldn’t tear themselves away from their computers. Fear was overblown, but made headlines. How was it overblown, we now have a non-trivial amount of completely de-socialized men in particular who live in online cults with real world impact. If there's one lesson from the last few decades it is that the people who were concerned about the impact of mass media on intelligence, physi…

That's fair. You are correct on potential for addiction. But let's be honest - most of these people, the ones the article is taking about, where they think they are some messiah, would have just latched onto some pre-internet cult regardless. Where sycophancy and love bombing was perfected. Though I do see the problem of AI assistants being much more accessible, so likely many more drawn in. https://en.wikipedia.org/…

> But let's be honest - most of these people, the ones the article is taking about, where they think they are some messiah, would have just latched onto some pre-internet cult regardless.

Honestly, I believe most people like this would just end up having a few odd beliefs that don't impact their ability to function or socialize, or at most, will get involved with some spiritual woo.

Such beliefs are compatible with American New Age spiritualism, for example. I've met a few spiritual people who have echoed the "I/we/you are god" sentiment, yet never lost their minds over it or joined cults.

I would not be surprised that if they were expertly manipulated by some of the most powerful AI models on this planet, they too, could be driven insane.

Re: People are losing loved ones to AI-fueled spiritual fantasies

#87

Post by manfromchina1 »

stevage wrote: post #21

Fascinating and terrifying. The allegations that ChatGPT is not discarding memory as requested are particularly interesting, wonder if anyone else has experienced this.

Grok was much more aggressive with this. It would constantly bring up what you said in the past with a date in parens. I dont see that anymore. > In the context of what you said about math(4/1/25) I think...

Re: People are losing loved ones to AI-fueled spiritual fantasies

#88

Post by kayodelycaon »

lamename wrote: post #74

If a Google engineer can get tricked by this, of course random people can. We're all human, including the flaws.

I agree.

The problem with expertise (or intelligence) is people think it’s transitive or applicable when it’s not.

At the end of the day, most people are just people.

Re: People are losing loved ones to AI-fueled spiritual fantasies

#89

Post by zmgsabst »

westurner wrote: post #16
MontagFTB wrote: post #15

Have we invited Wormwood to counsel us? To speak misdirected or even malignant advice that we readily absorb?

An LLM trained on all other science before Copernicus or Galileo would be expected to explain as true that the world is the flat center of the universe.

Ancient Greeks suspected it was spherical and had an estimated size accurate within 10%.

https://en.wikipedia.org/wiki/Eratosthenes

Re: People are losing loved ones to AI-fueled spiritual fantasies

#90

Post by EigenLord »

gngoo wrote: post #39

Working on AI myself, creating small and big systems, creating my own assistants and side-kicks. And then also seeing progress as well as rewards. I realize that I am not immune to this. Even when I am fully aware, I still have a feeling that some day I just hit the right buttons, the right prompts, and what comes staring back to me is something of my own creation that others see as some "fantasy" that I can't steer…

Years ago, in my writings I talked about the dangers of "oracularizing AI". From the perspective of those who don't know better, the breadth of what these models have memorized begins to approximate omniscience. They don't realize that LLMs don't actually truly know anything, there is no subject of knowledge that experiences knowing on their end. ChatGPT can speak however many languages, write however many programming languages, give lessons on virtually any topic that is part of humanity's general knowledge. If you attribute a deeper understanding to that memorization capability I can see how it would throw someone through a loop.

At the same time, there is quite a demand for a (somewhat) neutral, objective observer to look at our lives outside the morass of human stakes. AI's status as a nonparticipant, as a deathless, sleepless observer, makes it uniquely appealing and special from an epistemological standpoint. There are times when I genuinely do value AI's opinion. Issues with sycophancy and bias obviously warrant skepticism. But the desire for an observer outside of time and space persists. It reminds me of a quote attributed to Voltaire: "If God didn't exist it would be necessary to invent him."

Post reply on HN