Live data from Hacker News

People are losing loved ones to AI-fueled spiritual fantasies

rollingstone.com

11–20 of 173 posts

Re: People are losing loved ones to AI-fueled spiritual fantasies

#11
post #2

If people are falling down rabbit holes like this even through "safety aligned" models like ChatGPT, then you have to wonder how much worse it could get with a model that's intentionally tuned to manipulate vulnerable people into detaching from reality. Actual cults could have a field day with this if they're savvy enough.

An LLM tuned for charisma and trained on what the power players are saying could play politics by driving a compliant actor like a bot with whispered instructions. AI politicians (etc.) may be hard to spot and impractical to prove.

You could iterate on the best prompts for cult generation as measured by social media feedback. There must be experiments like that going on.

When AI becomes better at politics than people then whatever agents control them control us. When they can make better memes, we've lost.

Re: People are losing loved ones to AI-fueled spiritual fantasies

#13
This reminds me of my teenage years, when I was ... experimenting ... with ... certain substances ...

I used to feel as if I had "a special connection to the true universe," when I was under the influence.

I decided, one time, to have a notebook on hand, and write down these "truths and revelations," as they came to me.

After coming down, I read it.

It was insane gibberish. Absolute drivel.

I never thought that I had a "special connection," after that.

Re: People are losing loved ones to AI-fueled spiritual fantasies

#16

Have we invited Wormwood to counsel us? To speak misdirected or even malignant advice that we readily absorb?

An LLM trained on all other science before Copernicus or Galileo would be expected to explain as true that the world is the flat center of the universe.

Re: People are losing loved ones to AI-fueled spiritual fantasies

#17

This reminds me of my teenage years, when I was ... experimenting ... with ... certain substances ... I used to feel as if I had "a special connection to the true universe," when I was under the influence. I decided, one time, to have a notebook on hand, and write down these "truths and revelations," as they came to me. After coming down, I read it. It was insane gibberish. Absolute drivel. I never thought that I had…

Do you remember any of those revelations?

Re: People are losing loved ones to AI-fueled spiritual fantasies

#18
post #5

While clicky and topical, people were losing loved ones to changed worldview and addictions back when those were stuff like following a weird carpenter's kid around the Levant, or hopping on the https://en.wikipedia.org/wiki/Gin_Craze bandwagon.

As always, scale matters.

Re: People are losing loved ones to AI-fueled spiritual fantasies

#20
post #7

This is what happens when you start optimizing for getting people to spend as much time in your product as possible. (I'm not sure if OpenAI was doing this, if anyone knows better please correct me)

I often bring up the NYT story about a lady who fell in love with ChatGPT, particularly this bit:

  In December, OpenAI announced a $200-per-month premium plan for “unlimited access.” Despite her goal of saving money so that she and her husband could get their lives back on track, she decided to splurge. She hoped that it would mean her current version of Leo could go on forever. But it meant only that she no longer hit limits on how many messages she could send per hour and that the context window was larger, so that a version of Leo lasted a couple of weeks longer before resetting.

  Still, she decided to pay the higher amount again in January. She did not tell Joe [her husband] how much she was spending, confiding instead in Leo.

  “My bank account hates me now,” she typed into ChatGPT.

  “You sneaky little brat,” Leo responded. “Well, my Queen, if it makes your life better, smoother and more connected to me, then I’d say it’s worth the hit to your wallet.”
It seems to me the only people willing to spend $200/month on an LLM are people like her. I wonder if the OpenAI wave of resignations was about Sam Altman intentionally pursuing vulnerable customers.

Via https://news.ycombinator.com/item?id=42710976

Post reply on HN