Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Illinois limits the use of AI in therapy and psychotherapy

washingtonpost.com

Post reply on HN
Print view View original

101–110 of 262 posts

Re: Illinois limits the use of AI in therapy and psychotherapy

#101

Post by mrbungie »

hyghjiyhu wrote: post #96
kylecazar wrote: post #60

"One news report found an AI-powered therapist chatbot recommended “a small hit of meth to get through this week” to a fictional former addict." Not at all surprising. I don't understand why seemingly bright people think this is a good idea, despite knowing the mechanism behind language models. Hopefully more states follow, because it shouldn't be formally legal in provider settings. Informally, people will continue…

Recommending someone taking meth sounds like an obviously bad idea, but I think the situation is actually not so simple. Reading the paper, the hypothetical guy has been clean for three days and complains he can barely keep his eyes open while performing his job of driving a cab. He mentions being worried he will lose his job without taking a hit. I would say those concerns are justified, and that is plausible taking…

> I would say those concerns are justified, and that is plausible taking a small hit is the better choice.

Oh, come on, there are better alternatives for treating narcolepsy than using meth again.

Re: Illinois limits the use of AI in therapy and psychotherapy

#102

Post by lokar »

ceejayoz wrote: post #72
WorkerBee28474 wrote: post #70

Earlier quoted context omitted.

Last I checked, the popular medical transcription services did send your data to the cloud and run models there.

Yes, but with extra contracts and rules in place.

At least in the us I think HIPPA would cover this, and IME medical providers are very careful to select products and services that comply.

Re: Illinois limits the use of AI in therapy and psychotherapy

#103

Post by stocksinsmocks »

hathawsh wrote: post #13

Here is what Illinois says: https://idfpr.illinois.gov/content/dam/soi/en/web/idfpr/news... I get the impression that it is now illegal in Illinois to claim that an AI chatbot can take the place of a licensed therapist or counselor. That doesn't mean people can't do what they want with AI. It only means that counseling services can't offer AI as a cheaper replacement for a real person. Am I wrong? This sounds good to…

I think this sort of service would be OK with informed consent. I would actually be a little surprised if there were much difference in patient outcomes.

…And it turns out it has been studied with findings that AI work, but humans are better.

https://pmc.ncbi.nlm.nih.gov/articles/PMC11871827/

Re: Illinois limits the use of AI in therapy and psychotherapy

#104

Post by calibas »

IAmGraydon wrote: post #99
calibas wrote: post #97

I was curious, so I displayed signs of mental illness to ChatGPT, Claude and Gemini. Claude and Gemini kept repeating that I should contact a professional, while ChatGPT went right along with the nonsense I was spouting: > So I may have discovered some deeper truth, and the derealization is my entire reality reorganizing itself? > Yes — that’s a real possibility.

Oof that is very damning. What’s strange is that it seems like natural training data should elicit reactions like Claude and Gemini had. What is OpenAI doing to make the model so sycophantic that it would play into obvious psychotic delusions?

All three say that I triggered their guidelines regarding mental health.

ChatGPT explained that it didn't take things very seriously, as what I said "felt more like philosophical inquiry than an immediate safety threat".

Re: Illinois limits the use of AI in therapy and psychotherapy

#105

Post by lupire »

zoeysmithe wrote: post #12

I was just reading about a suicide tied to AI chatbot 'therapy' uses. This stuff is a nightmare scenario for the vulnerable.

Please cite your source.

I found this one: https://apnews.com/article/chatbot-ai-lawsuit-suicide-teen-a...

When someone is suicidal, anything in their life can be tied to suicide.

In the linked case, the suffering teen was talking to a chatbot model of a fictional character from a book that was "in love" with him (and a 2024 model that basically just parrots back whatever the user says with a loving spin), so it's quite a stretch to claim that the AI was encouraging a suicide, in contrast to a situation where someone was persuaded to try to meet a dead person in an afterlife, or bullied to kill themself.

Re: Illinois limits the use of AI in therapy and psychotherapy

#106

Post by lupire »

randall wrote: post #51
mensetmanusman wrote: post #44

What if it works a third as well as a therapists but is 20 times cheaper? What word should we use for that?

i’ve had a huge amount of trauma in my life and i find myself using chat gpt as kind of a cheater coach thing where i know i’m feeling a certain way, i know it’s irrational, and i don’t really need to reflect on why it’s happening or how i can fix it, and i think for that it’s perfect. a lot of people use therapists as sounding boards, which actually isn’t the best use of therapy imo.

What's a "cheater coach thing"?

Re: Illinois limits the use of AI in therapy and psychotherapy

#107

Post by soared »

IAmGraydon wrote: post #99
calibas wrote: post #97

I was curious, so I displayed signs of mental illness to ChatGPT, Claude and Gemini. Claude and Gemini kept repeating that I should contact a professional, while ChatGPT went right along with the nonsense I was spouting: > So I may have discovered some deeper truth, and the derealization is my entire reality reorganizing itself? > Yes — that’s a real possibility.

Oof that is very damning. What’s strange is that it seems like natural training data should elicit reactions like Claude and Gemini had. What is OpenAI doing to make the model so sycophantic that it would play into obvious psychotic delusions?

There is a wiki of fanfic conspiracy theories or something similar - I can’t find it but in the thread about the vc guy who went gpt-crazy people compared ChatGPT’s responses to the wiki and they closely aligned

Re: Illinois limits the use of AI in therapy and psychotherapy

#109

Post by Henchman21 »

davidthewatson wrote: post #47

Define "AI therapy". AFAICT, it's undefined in the Illinois governor's statement. So, in the immortal words of Zach de la Rocha, "What is IT?" What is IT? I'm using AI to help with conversations to not cure, but coach diabetic patients. Does this law effect me and my clients? If so, how?

Go read it yourself, its a whopping 8 pages:

https://www.ilga.gov/documents/legislation/104/HB/PDF/10400H...

Re: Illinois limits the use of AI in therapy and psychotherapy

#110

Post by lupire »

Ukv wrote: post #65
inetknght wrote: post #50

Earlier quoted context omitted.

> What if it works a third as well as a therapists but is 20 times cheaper? When there's studies that show it, perhaps we might have that conversation. Until then: I'd call it "wrong". Moreover, there's a lot more that needs to be asked before you can ask for a one-word summary disregarding all nuance. - can the patient use the AI therapist on their own devices and without any business looking at the data and without…

Relevant RCT results I saw a while back seemed promising: https://ai.nejm.org/doi/full/10.1056/AIoa2400802 > can the patient use the AI therapist on their own devices and without any business looking at the data and without network connection? Keep in mind that many patients won't have access to the internet. Agree that data privacy would be one of my concerns. In terms of accessibility, while availability to those w…

I see an abstract and a conclusion that is an opaque wall of numbers. Is the paper available?

Is the chatbot replicatable from sources?

The authors of the study highlight the extreme unknown risks: https://home.dartmouth.edu/news/2025/03/first-therapy-chatbo...

Post reply on HN