Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT terms disallow its use in providing legal and medical advice to others

ctvnews.ca

Post reply on HN
Print view View original

51–60 of 439 posts

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#51

Post by schiffern »

rafaelmn wrote: post #38
cpfohl wrote: post #3

Tricky…my son had a really rare congenital issue that no one could solve for a long time. After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth. I’m not saying we should be getting AI advice without a professional, but I’m my case it could have saved my kid a LOT of physical…

>After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth. Something I've noticed is that it's much easier to lead the LLM to the answer when you know where you want to go (even when the answer is factually wrong !), it doesn't have to be obvious leading but just framing the qu…

  >checking my now diagnosed gastritis I got stuff from GERD to CRC depending on which symptoms I decide to stress and which events I emphasize in the history
So... exactly the same behavior as human doctors?

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#52

Post by mirabilis »

entropicdrifter wrote: post #31
o11c wrote: post #25

When OpenAI is done getting rid of all the cases where its AI gives dangerously wrong advice about licensed professions, all that will be left is the cases where its AI gives dangerously wrong advice about unlicensed professions.

[flagged]

An AI-related bromide poisoning incident earlier this year: “Inspired by his history of studying nutrition in college, he decided to conduct a personal experiment to eliminate chloride from his diet. For 3 months, he had replaced sodium chloride with sodium bromide obtained from the internet after consultation with ChatGPT, in which he had read that chloride can be swapped with bromide, though likely for other purposes, such as cleaning… However, when we asked ChatGPT 3.5 what chloride can be replaced with, we also produced a response that included bromide. Though the reply stated that context matters, it did not provide a specific health warning, nor did it inquire about why we wanted to know, as we presume a medical professional would do.”

https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#53

Post by miltonlost »

blibble wrote: post #29
miltonlost wrote: post #27

Earlier quoted context omitted.

Sorry but you’re not gonna get me to agree that medical licensing is a bad idea. I don’t want quacks more than we already do. Stick to the argument and not add in your “what about” software engineers.

I am being serious... the damage certain software engineers could do certainly surpasses most doctors

Ah sorry, I misread it as coming from someone who doesn't want licensing, so you were appealing to HN by switching to software engineers (and I know many on here loathe to think anything beyond "move fast and break things", which is the opposite of most (non-software) engineers.

But yeah, I'd be down for at least some code of ethics, so we could have "do no harm" instead of "experiment on the mental states of children/adolescents/adults via algorithms and then do whatever is most addictive"

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#54

Post by fluidcruft »

Zr01 wrote: post #28

The cynic in me thinks this is just a means to eventually make more money by offering paid unrestricted versions to medical and legal professionals. I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence. Yet the same goes for just about any internet search. I don't think some people not knowing how to use it warrants restric…

I think one of the challenges is attribution. For example if you use Google search to create a fraudulent legal filing there aren't any of Google's fingerprints on the document. It gets reported as malpractice. Whereas reporting on these things is OpenAI or whatever AI is responsible. So even from the perspective of protecting a brand it's unavoidable. Suppose (no idea if true) the Louvre robbers wore Nike shoes and the reporting were that Nike shoes were used to rob the Louvre and all anyone talks about is Nike and how people need to be careful about what they do wearing Nike shoes.

It's like newsrooms took the advice that passive voice is bad form so they inject OpenAI as the subject instead.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#55

Post by fny »

bamboozled wrote: post #5

I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical). Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have…

I mean... you do have to backfill around your drainage pipe, so it's not too far off. Frankly, if you Google the subject people misspeak about "backfilling pipes" too as if the target of the backfill is the pipe itself too not the trench. Garbage in, garbage out.

All the circumstances where ChatGPT has given me shoddy advice fall in three buckets:

1. The internet lacks information, so LLMs will invent answers

2. The internet disagrees, so LLMs sometimes pick some answer without being aware of the others

3. The internet is wrong, so LLMs spew the same nonsense

Knowledge from blue collar trades seems often to in those three buckets. For subjects in healthcare, on the other hand, there are rooms worth of peer reviewed research, textbooks, meta studies, and official sources.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#56

Post by segmondy »

Zr01 wrote: post #28

The cynic in me thinks this is just a means to eventually make more money by offering paid unrestricted versions to medical and legal professionals. I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence. Yet the same goes for just about any internet search. I don't think some people not knowing how to use it warrants restric…

Nah, this is to avoid litigation. Who needs lawsuits when you are seeking profit? 1 loss of a major lawsuit is horrible, there's the case of folks suing them because their loved ones committed suicide after chatting with ChatGPT. They are doing everything to avoid getting dragged to court.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#57

Post by OutOfHere »

trollbridge wrote: post #22
PragmaCode wrote: post #15

It's not stopping to give legal/medical advice to the user, but it's forbidden to use ChatGPT to pose as an advisor giving advice to others: https://www.tomsguide.com/ai/chatgpt/chatgpt-will-still-offe...

One wonders how exactly this will be enforced.

It was already enforced by hiding all custom GPTs that offered medical advice.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#59

Post by Zr01 »

miltonlost wrote: post #45
Zr01 wrote: post #28

The cynic in me thinks this is just a means to eventually make more money by offering paid unrestricted versions to medical and legal professionals. I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence. Yet the same goes for just about any internet search. I don't think some people not knowing how to use it warrants restric…

> I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence. You are, but that's not how AI is being marketed by OpenAI, Google, etc. They never mention, in their ads, how much the output needs to be double and triple checked. They say "AI can do what you want! It knows all! It's smarter than PhDs!". Search engines don't say "An…

I appreciate how the newer versions provide more links and references. It makes the task of verifying it (or at least where it got its results from) that much easier. What you're describing seems more like a advertisement problem, not a product problem. No matter how many locks and restrictions they put on it, someone, somwhere, will still find a way to get hurt from its advice. A hammer that's hard enough to beat nails is hard enough to bruise your fingers.
Post reply on HN