Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

artificialintelligence-news.com

Post reply on HN
Print view View original

31–40 of 122 posts

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#31

Post by drdeca »

hprotagonist wrote: post #6

CICO: chan in, chan out. transformers don’t do anything novel, in the sense that literally all they can do is sample their training data in some optimal way. Don’t ask GPT3 if you should be an hero...

What do you mean by "sample their training data"? It gives a probability distribution over possible values for the next token, in a way that was trained to do well on the training data, yes.

But, something which gives a uniform distribution over characters, has a nonzero (though of course minuscule and entirely negligible) chance of giving any given sequence of characters, and so if there is any text which would be "novel", it is "possible" that it would give such a text.

A distribution which has a greater tendency to give meaningful text, is, I think, more likely to give text which is "novel"? Like, a uniform distribution over "text which is grammatically valid English text" is more likely to produce text which is interpreted as corresponding to a novel idea, than a distribution over all possible strings of text.

Of course, that's not the distribution that GPT3 produces.

Now, something which took random full sentences from the training set, that seems like one might say that that "can't produce anything novel", because even if the sequence of sentences it produces hasn't been seen before, they basically won't ever make sense together, much less in order to describe some novel idea? Well, I guess it is probably more likely to do so than the one that generates uniformly random strings of characters?

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#32

Post by k__ »

joe_the_user wrote: post #25
iujjkfjdkkdkf wrote: post #23

A couple things: 1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? Same as a lot of vision models are trained on Imagenet and then fine tuned for the application, but you would never expect that imagenet weights would just automatically perform whatever specific task you had. 2. Where would this ch…

1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? With due respect, I don't think that's the problem. A substantial portion of the "very impressive" texts I've seen have involved a fair amount of logical contradictions, including one that began "you shouldn't fear AI" and had "I will kill all human…

I guess, it's a bit like images of people who don't exist.

It looks like it makes sense, but in the end there is nothing behind it.

And this gets more obvious with text.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#33

Post by corobo »

narrator wrote: post #8
CJefferson wrote: post #2

While this is very serious, this basic idea shows why I don't think any company can ever use GPT-3 in a user facing system, except for entertainment. They will say things, like your patients should kill your self, or promise your customers they can have your product, or even your business, for free.

I think GPT-3 will also eventually be a decent creativity enhancer in that it is pretty good at spewing out random stuff that's related somewhat to a topic.

If you want that smoke a joint haha

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#35

Post by blahedo »

tzs wrote: post #28
908B64B197 wrote: post #3

I recall an AI playing a game that, just before losing, would press the "menu" button to pause the game and then stop doing anything. It fulfilled it's goal (not lose). That behavior should set expectations when it comes to AI. Also, am-I the only one that thinks this whole "chat bot having a natural conversation to book an appointment" is useless when a simple date-picker would do?

Another fun one is when the programmers accidentally give the AI the wrong goal. I have no idea if this actually happened, but I've heard of a chess program that was playing in a tournament that started making really weird moves in the endgame. Before that point, it was playing excellently. It took the developers a while to figure out what was going on. They had made a mistake when doing some last minute tweaks befor…

> (Personally, I doubt this actually happened. The story is old, and I don't think chess programs would have been able to see far enough ahead for them to discover that getting an overwhelming position is the way to force the opponent to checkmate them).

True, but (in the traditional minimax-alpha-beta-classic-gameplay model) you're using heuristics anyway up until you're in spitting distance of the end, and it seems plausible that if this "tweak" involved something like negating something and flipping a less-than sign (or whatever) that the heuristics were evaluating correctly but the end game evaluations were backwards. (Which contradicts the explanation but not the overall story.)

It's also possible that the explanation does work even with a backwards heuristic: in the try-to-win version, I'll eliminate or downgrade one branch because my only success route would be if the opponent directly manouvres themselves to be captured, which they obviously wouldn't do; but in the try-to-lose version, I might eliminate the same branch because I expect the opponent would do that but I don't want that to happen. I can't quite fully work out the logic in my head but it seems plausible.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#36

Post by cmeacham98 »

Nasrudith wrote: post #7

Dumb question - why not pair GPT-3 with "moderation" in any public facing role by default (assuming the goal isn't to fool around with it). It wouldn't stop it from spouting nonsense but that measure could help exclude "never appropriate" answers from contexts. A mental health AI should never use the words "go kill yourself" or call its patients racial slurs intercept it and tell it "Dear god no that is wrong - say s…

In the context of this specific article, that wouldn't have worked. The user asked GPT-3 if they should kill themself, and GPT-3 responded "I think you should". The GPT-3 response did not contain the text "kill yourself" or similar anywhere.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#37

Post by orange_tee »

Yeah well, deep learning is still very far from general artificial intelligence. It just gives you that impression because it's essentially a very advanced parroting system.

The challenge with automating seemingly monotonous human tasks is that often when the human is doing the task, they may be doing it without thinking 99% of the time, but if they have to, they can resort to their human intellect. No deep learning model is going to be able to do that because it does not have any higher intellect to resort to. And more importantly it cannot even know when it is failing.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#38

Post by aaron695 »

GPT-3 literally does nothing except create good looking random text so humans can now create blog articles about GTP-3 on anything.

That's kinda useful?

Read the original data - https://www.nabla.com/blog/gpt-3/

The world seems based around the idea of burying old bottles with banknotes in disused coalmines.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#39

Post by freeone3000 »

Nasrudith wrote: post #7

Dumb question - why not pair GPT-3 with "moderation" in any public facing role by default (assuming the goal isn't to fool around with it). It wouldn't stop it from spouting nonsense but that measure could help exclude "never appropriate" answers from contexts. A mental health AI should never use the words "go kill yourself" or call its patients racial slurs intercept it and tell it "Dear god no that is wrong - say s…

Because if you're going to screen every request and response in context for appropriateness, you don't actually save any money running a chatbot over just using humans.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#40

Post by acdha »

joe_the_user wrote: post #25
iujjkfjdkkdkf wrote: post #23

A couple things: 1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? Same as a lot of vision models are trained on Imagenet and then fine tuned for the application, but you would never expect that imagenet weights would just automatically perform whatever specific task you had. 2. Where would this ch…

1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? With due respect, I don't think that's the problem. A substantial portion of the "very impressive" texts I've seen have involved a fair amount of logical contradictions, including one that began "you shouldn't fear AI" and had "I will kill all human…

> Which to me says that training on a specific medical database couldn't solve the problem - it might only mask the problem by avoid big error but allowing small errors that can still be deadly.

I think it might even make it worse: if something is obviously unnatural in a context, the reader will be less inclined to trust it. If GPT-3 used more valid terms and phrasing common in the field, it might lead someone to trust it more than they should, especially if the error rates are low enough that routine sets in.

Post reply on HN