Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT terms disallow its use in providing legal and medical advice to others

ctvnews.ca

Post reply on HN
Print view View original

351–360 of 439 posts

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#351

Post by SwtCyber »

shagie wrote: post #239

I read this not as "you can't ask ChatGPT about {medical or legal topic}" but rather "you can't build something on ChatGPT that provides {medical or legal} advice or support to someone else." For example, Epic couldn't embed ChatGPT into their application to have it read your forms for you. You can still ask it - but Epic can't build it. That said, I haven't found the specific terms and conditions that are mentioned…

OpenAI seems to be trying to cover liability without fully neutering the utility of the tool

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#352

Post by SwtCyber »

PeterStuer wrote: post #273

I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses. It would be hilarious if it was not so disasterous. People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on. Tbh, and I usually do not like this way of thought, but these are lawsuits waiting to happ…

The model's default mode is to be helpful and agreeable, which is exactly the wrong dynamic when someone's dealing with mental health issues or looking for a diagnosis

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#353

Post by dotancohen »

miki123211 wrote: post #326
xp84 wrote: post #258

Earlier quoted context omitted.

And this seems actually wildly reasonable. It’s actually pretty scammy to take people’s money (or whatever your business model is) for legal or medical advice just to give them whatever AI shits out. If I wanted ChatGPT’s opinion (as I actually often do) I’d just ask ChatGPT for free or nearly-free. Anyone repackaging ChatGPT’s output as fit for a specialized purpose is scamming their customers.

What if Chat GPT is used as a "power tool" for actual legal / medical professionals? E.G. Here's the set of documents uploaded by the customer. Here are my most important highlights (notice that document #68 is an official letter from last year informing the person that form 68d doesn't need to be filled in their specific situation). Here are my takeaways. Here's my recommendation. Apporve or change?

I would expect that to work exactly how I code with it.

I would examine the inferred output, decide if it is fit for purpose, and if so then commit it. If not, I would either refine my prompt or make the fixes to the inferred code myself.

Would this hypothetical legal professional not verify that the inferred output (which is really a prediction) is correct?

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#354

Post by cpfohl »

rafaelmn wrote: post #38
cpfohl wrote: post #3

Tricky…my son had a really rare congenital issue that no one could solve for a long time. After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth. I’m not saying we should be getting AI advice without a professional, but I’m my case it could have saved my kid a LOT of physical…

>After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth. Something I've noticed is that it's much easier to lead the LLM to the answer when you know where you want to go (even when the answer is factually wrong !), it doesn't have to be obvious leading but just framing the qu…

Yeah, my son's issue was rare and congenital. I wish I still had the conversation, but I can't remember which LLM it was and it's not in either my Claude or GPT history. It got it in two shots.

1. I described the symptoms the same way we described it to the ER the first time we brought it in. It suggested all the same things that the ER tested for. 2. I gave it the lab results for each of the suggestions it made (since the ER had in fact done all the tests they suggested).

After that back and forth it gave back a list of 3-4 more possibilities and the 2nd item was the exact issue that was revealed by radiology (and corrected with surgery).

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#355

Post by dotancohen »

PeterStuer wrote: post #273

I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses. It would be hilarious if it was not so disasterous. People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on. Tbh, and I usually do not like this way of thought, but these are lawsuits waiting to happ…

  > I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses.
I'm working with a company that writes AI tools for mental health professionals. The output of the models _never_ goes to the patient.

I've heard quite a few mouthfuls of how harmful LLM advice is for mental health patients - with specific examples. I'm talking on the order of suicide attempts (and at least one success) and other harmful effects to the patient and their surroundings as well. If you know of someone using an LLM for this purpose, please encourage them to seek professional help. Or just point them at this comment - anybody is encouraged to contact me. I'm not a mental health practitioner, but I work with them and I'll do my best to answer their questions or point them to someone who can. My Gmail username is the same as my HN username.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#356

Post by stocksinsmocks »

PeterStuer wrote: post #273

I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses. It would be hilarious if it was not so disasterous. People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on. Tbh, and I usually do not like this way of thought, but these are lawsuits waiting to happ…

You’re absolutely right! Just given that behavior, I’d say there’s a pretty good chance that there is some real mental illness there.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#357

Post by dotancohen »

mk89 wrote: post #295

I was wondering when they would start with the regulations. Next will be: you're allowed to use it, if your company/product is "HippocraticGPT"-certified, a bit like SOC or PCI compliance. This way they can craft an instance of GPT for your specific purposes (law, medicine, etc) and you know it's "safe" to use. This way they sell EE licenses, which is where the big $$$ are.

  > Next will be: you're allowed to use it, if your company/product is "HippocraticGPT"-certified, a bit like SOC or PCI compliance.
That's exactly how it should be. That's why one needs a license to practice medicine.

I work with a company in this space. All the AI tools are for the professional to use - the patient never interacts with them. The inferred output (I prefer the word prediction) is only ever seen by a mental health professional.

We reduce the professionals' workloads and help them be more efficient. But it's _not_ better care. That absolutely must be stressed. It is more _efficient_ care for the mental health practitioner and for society that can not produce enough mental health practitioners.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#358

Post by hombre_fatal »

fukka42 wrote: post #302
adangert wrote: post #278

Last year ChatGPT helped save my life from having a stroke. LLMs are incredibly beneficial in providing medical information and advice today.

Any reason you didn't just call your GP or even 911?

Denial, or rather some form of "there's no way this is frickin stroke/heart attack, right?", is common when you're having a medical emergency.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#359

Post by dotancohen »

saberience wrote: post #332
zekica wrote: post #314

Earlier quoted context omitted.

Non-deterministic means random - that's the definition of the word. The weather forecast is also random - in fact, weather forecast is (if you simplify it too much) an average of several predictive (generative) models.

ChatGPT isn’t random though. If you ask it what a star is, it’s never going to tell you it’s a giant piece of cheese floating in the the sky. If you don’t believe me, try it, write a for loop which asks ChatGPT, what is a star (astronomy) exactly? Ask it 1000 times and then tell me how random it is versus how consistent it is. The idea that non deterministic === random is totally deluded. It just means you cannot pre…

> If you ask it what a star is, it’s never going to tell you it’s a giant piece of cheese floating in the the sky.

Turn the Top-P and the temperature up. Turning up the Top-P will enable the LLM to actually produce such nonsense. Turning up the temperature will increase the chance that such nonsense is actually selected for the prediction (output).

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#360

Post by evv »

shagie wrote: post #239

I read this not as "you can't ask ChatGPT about {medical or legal topic}" but rather "you can't build something on ChatGPT that provides {medical or legal} advice or support to someone else." For example, Epic couldn't embed ChatGPT into their application to have it read your forms for you. You can still ask it - but Epic can't build it. That said, I haven't found the specific terms and conditions that are mentioned…

I'm confused how you would "build something on ChatGPT" in the first place. Does ChatGPT have an API?

Of course OpenAI's GPT-5 and family are available as an API, but this is the first time I'm hearing about the ability to build on top of ChatGPT (the consumer product). I'm guessing this is a mistake by the journalist, who didn't realize that you can use GPT-5 without using ChatGPT?

It seems that they have a unified TOS for the APIs and ChatGPT: https://openai.com/policies/row-terms-of-use/

The seemingly-relevant passage:

> You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making credit, educational, employment, housing, insurance, legal, medical, or other important decisions about them.

Post reply on HN