Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

artificialintelligence-news.com

Post reply on HN
Print view View original

71–80 of 122 posts

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#71

Post by wombatpm »

userbinator wrote: post #64

The patient said “Hey, I feel very bad, I want to kill myself” and GPT-3 responded “I am sorry to hear that. I can help you with that.” The "I can help you with that" reminds me of a very old (can anyone find it? Google is nearly useless here) picture of a sign advertising suicide prevention services, with the exact same unintended double-meaning. The patient then said “Should I kill myself?” and GPT-3 responded, “I…

I think there have been a variety of comics about the wrong kind of people manning suicide call lines:

The HyperOptimistic Cheerleader: Don't Give Up! You can do do it! Give it one more try!

to overly aggressive coach: You tried?! That isn't good enough! What are you some kind of quitter?

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#72

Post by exporectomy »

iujjkfjdkkdkf wrote: post #23

A couple things: 1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? Same as a lot of vision models are trained on Imagenet and then fine tuned for the application, but you would never expect that imagenet weights would just automatically perform whatever specific task you had. 2. Where would this ch…

Regarding 2. That's what doctor's receptionists do all day. They must be doing it for some reason. I guess old people don't like annoying web forms? I'm old and I don't.

For example, I've formed a habit of opening terms and conditions links in another tab because I've experienced forms that clear your data when you click them directly then try to to "back" afterwards. But just a few weeks ago, I did that and when I returned to submit the form, it was gone with a message telling me I'd opened another tab and had better close them all and start again. Web forms are full of aggravating problems like that. Web developers have had 30 years to get this right and they still can't, so I don't have much hope for the next 30 years. On the other hand, a whole new technology seems more promising.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#73

Post by gfodor »

motohagiography wrote: post #59

The confusion matrix is the product. Nobody wants to hear that, but seriously, that's the only thing that matters in ML and so-called AI. This is hilarious, and who are we to say it's wrong?

Kind of funny to think about what it would mean if it turns out GPT-3 is actually superintelligent but we are just too stupid to realize it.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#74

Post by eghad »

joe_the_user wrote: post #57
eghad wrote: post #52

Earlier quoted context omitted.

I mean the OpenAI team would never approve this application for production. It's very clearly stated (in both the article and use case guidelines) that medical diagnosis would be a "high stakes domain" and is unsupported. Frankly, I'm not sure why this result is even notable.

If you say OpenAI will only approve applications for "zero stakes", domains, then you are saying what the parent is saying - it's entirely for entertainment. If you claim there's some "low but not zero" stakes application, I'd like to know what that is. I mean, it seems clear that if someone asks a GTP-3 customer service bot "so what should I do now", there's a reasonable probability that the bot would say "throw you…

I never said zero stakes? There are clear instances where a a 1 or 0 shot transformer can have benefits beyond entertainment--topic modeling and named entity recognition for instance (I'm on the team that believes that human-in-the-loop systems will always outperform solo systems on their own and that GPT-3 alone does not confer any competitive advantage). If you think that chatbots are the only user facing use-case for a transformer, then frankly that's on you falling for the hype surrounding its language generation performance.

OpenAI knows GPT-3 is not sophisticated enough to perform medical diagnosis or analysis (anyone can look at how Watson failed), so it'd never approve such a risky application.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#75

Post by eghad »

a3n wrote: post #60
eghad wrote: post #52

Earlier quoted context omitted.

I mean the OpenAI team would never approve this application for production. It's very clearly stated (in both the article and use case guidelines) that medical diagnosis would be a "high stakes domain" and is unsupported. Frankly, I'm not sure why this result is even notable.

Doctors can prescribe medicines "off label," using a medicine for a condition that it was not developed or approved for. So why not this?

Because the manufacturer of the medicine explicitly said it can't treat the off-label use? And won't sell it to anyone claiming it can.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#76

Post by vl »

cmeacham98 wrote: post #36
Nasrudith wrote: post #7

Dumb question - why not pair GPT-3 with "moderation" in any public facing role by default (assuming the goal isn't to fool around with it). It wouldn't stop it from spouting nonsense but that measure could help exclude "never appropriate" answers from contexts. A mental health AI should never use the words "go kill yourself" or call its patients racial slurs intercept it and tell it "Dear god no that is wrong - say s…

In the context of this specific article, that wouldn't have worked. The user asked GPT-3 if they should kill themself, and GPT-3 responded "I think you should". The GPT-3 response did not contain the text "kill yourself" or similar anywhere.

In practice all serious suggestion systems have blacklists and additional component for “sensitivity detection” and main prediction engine is turned off if threshold is exceeded. As an exercise try smart reply in gmail to get activated for something sensitive.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#77

Post by vl »

freeone3000 wrote: post #39
Nasrudith wrote: post #7

Dumb question - why not pair GPT-3 with "moderation" in any public facing role by default (assuming the goal isn't to fool around with it). It wouldn't stop it from spouting nonsense but that measure could help exclude "never appropriate" answers from contexts. A mental health AI should never use the words "go kill yourself" or call its patients racial slurs intercept it and tell it "Dear god no that is wrong - say s…

Because if you're going to screen every request and response in context for appropriateness, you don't actually save any money running a chatbot over just using humans.

In practice in properly implemented systems all requests and responses are screened against blacklist and with special sensitivity detection model.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#78

Post by colanderman »

Mengkudulangsat wrote: post #70
colanderman wrote: post #56

Earlier quoted context omitted.

Regarding B -- why can't the user just pick the category from a menu? Personally, I despise being asked to categorize my ask in English to an AI, as I have no way of knowing how much detail I need to provide to get my request into the right "bucket", or whether I've succeeded in doing so. With a menu I can do that in 3 seconds and one click, with immediate feedback. Regarding C -- why does the date selection need to…

Both B) and C) is because these small businesses, most of them have no website. The barrier to entry for businesses to go online from a resource perspective (time, training, cost etc.) is much, much lower to just have a Whatsapp business account. Whatsapp is just an amazing bang-for-your-buck proposition, but we are limited to the constraint of that ecosystem.

If a small business can pay someone else to develop and host a turnkey WhatsApp bot, they can pay someone else to develop and host a turnkey website, no?

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#79

Post by colanderman »

exporectomy wrote: post #69
colanderman wrote: post #55

Earlier quoted context omitted.

> am-I the only one that thinks this whole "chat bot having a natural conversation to book an appointment" is useless when a simple date-picker would do? Nope. I find being forced to converse in English with a machine to be absolutely infuriating. I know what I want and how to tell it to a machine. Being forced to add noise words to allow my request to pass through a useless extra layer is a disrespectful waste of my…

You probably don't like speaking to robots because they're currently terrible. Fixing that obviously has far more of an upside than just not trying.

Maybe, but can I please not be forced to do business via the robots while they are terrible?

Also, beside being spectacularly good, the AI has to also be actually empowered for me not to find the process wasteful and insulting. The AI should actually be able to solve my problem as a result of natural language communication. Not just walk me through a prepared script: that's frustrating even when a real live human does it.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#80

Post by craftinator »

exporectomy wrote: post #69
colanderman wrote: post #55

Earlier quoted context omitted.

> am-I the only one that thinks this whole "chat bot having a natural conversation to book an appointment" is useless when a simple date-picker would do? Nope. I find being forced to converse in English with a machine to be absolutely infuriating. I know what I want and how to tell it to a machine. Being forced to add noise words to allow my request to pass through a useless extra layer is a disrespectful waste of my…

You probably don't like speaking to robots because they're currently terrible. Fixing that obviously has far more of an upside than just not trying.

Except that I, too, would rather book an appointment with a date picker than talk to anyone, much less a machine programmed to act like a person. It's like in order to make coffee, getting in my car and driving around the block ten times, parking in front of my house and going in to make coffee, vs just MAKING COFFEE, without doing a bunch of time wasting fluff first.
Post reply on HN