Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT terms disallow its use in providing legal and medical advice to others

ctvnews.ca

Post reply on HN
Print view View original

11–20 of 439 posts

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#11

Post by FaradayRotation »

bamboozled wrote: post #5

I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical). Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have…

I nearly spit my drink out. This is my kind of humor, thanks for sharing.

I've had a decent experience (though not perfect) with identifying and understanding building codes using both Claude and GPT. But I had to be reasonably skeptical and very specific to get to where I needed to go. I would say it helped me figure out the right questions and which parts of the code applied to my scenario, more than it gave the "right" answer the first go round.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#12

Post by miki123211 »

> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”

Is this an actual technical change, or just legal CYA?

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#13

Post by bamboozled »

dingnuts wrote: post #9
bamboozled wrote: post #5

I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical). Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have…

honestly I think these things cause a form of Gell Mann's Amnesia where when you use them for something you know already, the errors are obvious, but when you use them for something you don't understand already, the output is sufficiently plausible that you can't tell you're being misled. this makes the tool only useful for things you already know! I mean, just in this thread there's an anecdote from a guy who used i…

It’s funny you should say that because I have been using it in the way you describe. I kind of know it could be wrong, but I’m kind of desperate for info so I consult Claude anyway. After stressing hard I realize it was probably wrong, find someone who knows what they’re and actually on about and course correct.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#14

Post by rokkamokka »

dingnuts wrote: post #9
bamboozled wrote: post #5

I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical). Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have…

honestly I think these things cause a form of Gell Mann's Amnesia where when you use them for something you know already, the errors are obvious, but when you use them for something you don't understand already, the output is sufficiently plausible that you can't tell you're being misled. this makes the tool only useful for things you already know! I mean, just in this thread there's an anecdote from a guy who used i…

I'd frame it such that LLM advice is best when it's the type that can be quickly or easily confirmed. Like a pointer in the right (or wrong) direction. If it was false, then try again - quick iterations. Taking it at its "word" is the potentially harmful bit.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#16

Post by bearhall »

miki123211 wrote: post #12

> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.” Is this an actual technical change, or just legal CYA?

I think it actually changed. I have a broken bone and have been consulting with ChatGPT (along with my doctor of course) for the last week. Last night it refused to give an opinion saying “ While I can’t give a medical opinion or formally interpret it”. First time I’d seen it object.

I understand the change but it’s also a shame. It’s been a fantastically useful tool for talking through things and educating myself.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#17

Post by cj »

throwaway290 wrote: post #6
cpfohl wrote: post #3

Tricky…my son had a really rare congenital issue that no one could solve for a long time. After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth. I’m not saying we should be getting AI advice without a professional, but I’m my case it could have saved my kid a LOT of physical…

this is fresh news right? a friend just used chatgpt for medical advice last week (stuffed his wound with antibiotics after motorbike crash). are you saying you completely treated the congenital issue in this timeframe?

He’s simply saying that ChatGPT was able to point them in the right direction after 1 chat exchange, compared to doctors who couldn’t for a long time.

Edit: Not saying this is the case for the person above, but one thing that might bias these observations is ChatGPT’s memory features.

If you have a chat about the condition after it’s diagnosed, you can’t use the same ChatGPT account to test whether it could have diagnosed the same thing (since the chatGPT account now knows the son has a specific condition).

The memory features are awesome but also sucks at the same time. I feel myself getting stuck in a personalized bubble even more so than Google.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#18

Post by justapassenger »

bamboozled wrote: post #5

I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical). Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have…

I'm a hobby woodworker - I've tried using gemini recently for an advice on how to make some tricky cuts.

If I'd follow any of the suggestions I'd probably be in ER. Even after me pointing out issues and asking it to improve - it'd come up with more and more sophistical ways of doing same fundamentally dangerous actions.

LLMs are AMAZING tools, but they are just that - tools. There's no actual intelligence there. And the confidence with which they spew dangerous BS is stunning.

Re: ChatGPT terms disallow its use in providing legal and medical advice to others

#19

Post by blibble »

miltonlost wrote: post #10

Good. Techies need to stop thinking that an LLM should not be immune from requiring licensing. Until OpenAI can (and should) be sued for medical malpractice or lawyering without passing the bar, they will have no skin in the game to actually care. A disclaimer of "this is not a therapist" should not be enough to CYA.

anyone wanna form a software engineering guild, then lobby to need a license granted by the guild to practice?
Post reply on HN