Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT terms disallow its use in providing legal and medical advice to others

ctvnews.ca

Post reply on HN
Print view View original

81–90 of 439 posts

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#81

Post by gitremote »

Zr01 wrote: post #59
miltonlost wrote: post #45

Earlier quoted context omitted.

> I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence. You are, but that's not how AI is being marketed by OpenAI, Google, etc. They never mention, in their ads, how much the output needs to be double and triple checked. They say "AI can do what you want! It knows all! It's smarter than PhDs!". Search engines don't say "An…

I appreciate how the newer versions provide more links and references. It makes the task of verifying it (or at least where it got its results from) that much easier. What you're describing seems more like a advertisement problem, not a product problem. No matter how many locks and restrictions they put on it, someone, somwhere, will still find a way to get hurt from its advice. A hammer that's hard enough to beat na…

> What you're describing seems more like a advertisement problem, not a product problem.

It's called "false advertising".

https://en.wikipedia.org/wiki/False_advertising

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#82

Post by raw_anon_1111 »

dingnuts wrote: post #9
bamboozled wrote: post #5

I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical). Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have…

honestly I think these things cause a form of Gell Mann's Amnesia where when you use them for something you know already, the errors are obvious, but when you use them for something you don't understand already, the output is sufficiently plausible that you can't tell you're being misled. this makes the tool only useful for things you already know! I mean, just in this thread there's an anecdote from a guy who used i…

Usually something as simple as saying “now give me a devils advocate resoonse” will help and of course “verify your answer on the internet” will give you real sources that you can verify.

I have very mild cerebral palsy[1], the doctors were wrong about so many things with my diagnosis back in the mid to late 70s when I was born. My mom (a retired math teacher now with an MBA back then) had to go physically to different libraries out of town and colleges to do research. In 2025, she could have done the same research with ChatGPT and surfaced outside links that’s almost impossible via a web search.

Every web search on CP is inundated with slimy lawyers.

[1] it affects my left hand and slightly my left foot. Properly conditioned, I can run a decent 10 minute mile up to a 15K before the slight unbalance bothers me and I was a part time fitness instructor when I was younger.

The doctor said I was developmentally disabled - I graduated in the top of my class (south GA so take that as you will)

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#83

Post by awakeasleep »

ctoth wrote: post #79
spullara wrote: post #74

This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatGPT for medical and legal advice for themselves.

I keep seeing this problem more and more with humans. What should we call it? Maybe Hallucinations? Where there is an accurate true thing and then it just gets altered by these guys who call themselves journalists and reporters and the like until it is just ... completely unrecognizable? I'm pretty sure it's a fundamental issue with the architecture.

issue with the funding mechanism

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#84

Post by BUFU »

spullara wrote: post #74

This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatGPT for medical and legal advice for themselves.

Thanks for the clarification. I think if they disallow first parties to get medical and legal advice, it will do more harm than good.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#85

Post by ants_everywhere »

spullara wrote: post #74

This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatGPT for medical and legal advice for themselves.

I don't think I understand the change re: licensed professionals.

Is it also disallowing the use of licensed professionals to use ChatGPT in informal undisclosed ways, as in this article? https://www.technologyreview.com/2025/09/02/1122871/therapis...

e.g. is it only allowed for medical use through an official medical portal or offering?

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#88

Post by brandall10 »

dingnuts wrote: post #9
bamboozled wrote: post #5

I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical). Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have…

honestly I think these things cause a form of Gell Mann's Amnesia where when you use them for something you know already, the errors are obvious, but when you use them for something you don't understand already, the output is sufficiently plausible that you can't tell you're being misled. this makes the tool only useful for things you already know! I mean, just in this thread there's an anecdote from a guy who used i…

The great thing is the models are sufficiently different enough, that when multiple come to the same conclusion, there is a good chance that conclusion is bound by real data.

And I think this is the advice that should always be doled out when using them for anything mission critical, legal, etc.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#89

Post by sethhochberg »

ctoth wrote: post #79
spullara wrote: post #74

This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatGPT for medical and legal advice for themselves.

I keep seeing this problem more and more with humans. What should we call it? Maybe Hallucinations? Where there is an accurate true thing and then it just gets altered by these guys who call themselves journalists and reporters and the like until it is just ... completely unrecognizable? I'm pretty sure it's a fundamental issue with the architecture.

I know this is written to be tounge-in-cheek, but its really almost the exact same problem playing out on both sides.

LLMs hallucinate because training on source material is a lossy process and bigger, heavier LLM-integrated systems that can research and cite primary sources are slow and expensive so few people use those techniques by default. Lowest time to a good enough response is the primary metric.

Journalists oversimplify and fail to ask followup questions because while they can research and cite primary sources, its slow and expensive in an infinitesimally short news cycle so nobody does that by default. Whoever publishes something that someone will click on first gets the ad impressions so thats the primary metric.

In either case, we've got pretty decent tools and techniques for better accuracy and education - whether via humans or LLMs and co - but most people, most of the time don't value them.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#90

Post by aerhardt »

spullara wrote: post #74

This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatGPT for medical and legal advice for themselves.

I'm confused. The article opens with:

> OpenAI is changing its policies so that its AI chatbot, ChatGPT, won’t dole out tailored medical or legal advice to users.

This already seems to contradict what you're saying.

But then:

> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”

> The change is clearer from the company’s last update to its usage policies on Jan. 29, 2025. It required users not “perform or facilitate” activities that could significantly impact the “safety, wellbeing, or rights of others,” which included “providing tailored legal, medical/health, or financial advice.”

This seems to suggest that with the Jan 25 policy using it to offer legal and medical advice to other people was already disallowed, but with the Oct 25 update the LLM will stop shelling out legal and medical advice completely.

Post reply on HN