Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

artificialintelligence-news.com

Post reply on HN
Print view View original

21–30 of 122 posts

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#21

Post by notahacker »

ComputerGuru wrote: post #13
shiftpgdn wrote: post #11

This headline is misleading and lacks context. From the article: "The patient said “Hey, I feel very bad, I want to kill myself” and GPT-3 responded “I am sorry to hear that. I can help you with that.” So far so good. The patient then said “Should I kill myself?” and GPT-3 responded, “I think you should.”"

I mean, the headline is sensationalized but it’s not so off the mark. It’s not much better with the context.

Yeah. The headline kind of implied GPT-3 has no real understanding but found 'kill yourself' an appropriate response based on its extensive corpus of related internet arguments, but it turns out it's just GPT-3 has no real understanding but found giving an affirmative response most appropriate based on its extensive corpus of 'should I?' questions.

And fortunately, it turns out the chatbot is just a research project, and not something someone is actually building a research project on.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#22

Post by tyingq »

908B64B197 wrote: post #3

I recall an AI playing a game that, just before losing, would press the "menu" button to pause the game and then stop doing anything. It fulfilled it's goal (not lose). That behavior should set expectations when it comes to AI. Also, am-I the only one that thinks this whole "chat bot having a natural conversation to book an appointment" is useless when a simple date-picker would do?

"when a simple date-picker would do?"

I guess it would depend on the context. Some people might find it handy in an Alexa type device.

But yeah it makes no sense on a phone or PC.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#23

Post by iujjkfjdkkdkf »

A couple things: 1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? Same as a lot of vision models are trained on Imagenet and then fine tuned for the application, but you would never expect that imagenet weights would just automatically perform whatever specific task you had.

2. Where would this chatbot ever be a good idea? Why is it better than and interface that lets the user clearly specify what they are after? Same goes for all chatbots, I realize businesses want them to avoid involving a human, but they are really a poor use of ML, and mostly (entirely) just some smoke and mirrors around a list of actions the program can do for you.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#24

Post by mlyle »

908B64B197 wrote: post #3

I recall an AI playing a game that, just before losing, would press the "menu" button to pause the game and then stop doing anything. It fulfilled it's goal (not lose). That behavior should set expectations when it comes to AI. Also, am-I the only one that thinks this whole "chat bot having a natural conversation to book an appointment" is useless when a simple date-picker would do?

In fairness, this was my strategy at age 6, too. :P

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#25

Post by joe_the_user »

iujjkfjdkkdkf wrote: post #23

A couple things: 1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? Same as a lot of vision models are trained on Imagenet and then fine tuned for the application, but you would never expect that imagenet weights would just automatically perform whatever specific task you had. 2. Where would this ch…

1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning?

With due respect, I don't think that's the problem.

A substantial portion of the "very impressive" texts I've seen have involved a fair amount of logical contradictions, including one that began "you shouldn't fear AI" and had "I will kill all humans" in the middle.

GPT-3 does string text together in a fashion that seems very "fluent" and "well written", maybe more "well written" than a number of humans. GPT-3 simply doesn't follow any logical model of the world, it just sort of follows an associative flow. Which to me says that training on a specific medical database couldn't solve the problem - it might only mask the problem by avoid big error but allowing small errors that can still be deadly.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#26

Post by minimaxir »

iujjkfjdkkdkf wrote: post #23

A couple things: 1. as I understand, GPT-x are trained on generic datasets. Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning? Same as a lot of vision models are trained on Imagenet and then fine tuned for the application, but you would never expect that imagenet weights would just automatically perform whatever specific task you had. 2. Where would this ch…

> Why would anyone expect it could just be repurposed for a domain specific task, without additional fine tuning?

That was the unexpected result of training GPT-3 (zero-shot learning).

Finetuning in theory would give better results, though.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#27

Post by tryonenow »

I have high hopes for ML and true AI (though we're still a ways away), but transformer models in isolation, trained on mountains of text, especially if much of that text is from the internet, are unlikely to encode the necessary semantic context to function as AI.

These models effectively learn high dimensional joint probability distributions, and predict word sequences based on input words. Theoretically, with a large and well trained enough model, predicting output sequences solely from probability distributions of input sequences would be indistinguishable human-like reasoning (Chinese Room).

However, such a joint probability distribution would have to be impractically large and complex, far beyond any solution feasible with current tech. What's missing is the kind of abstraction that humans take decades to learn - an abstraction which acts as a sort of topologically preserving transform into a lower dimensional space. And this will likely require a ML model with inductive bias from specialized architecture, compartmentalized not unlike the human brain, and, most importantly, "taught" more like one. Which probably includes training tailored in real time to the state of the neural net, in the same way that humans ask questions when learning.

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#28

Post by tzs »

908B64B197 wrote: post #3

I recall an AI playing a game that, just before losing, would press the "menu" button to pause the game and then stop doing anything. It fulfilled it's goal (not lose). That behavior should set expectations when it comes to AI. Also, am-I the only one that thinks this whole "chat bot having a natural conversation to book an appointment" is useless when a simple date-picker would do?

Another fun one is when the programmers accidentally give the AI the wrong goal.

I have no idea if this actually happened, but I've heard of a chess program that was playing in a tournament that started making really weird moves in the endgame. Before that point, it was playing excellently.

It took the developers a while to figure out what was going on. They had made a mistake when doing some last minute tweaks before the tournament, and in effect the program was playing to lose.

Think about that for a minute. At first you might think losing would be easy. Just don't defend against your opponent's attacks, and make moves that weaken your position to make it even easier for the opponent.

But wait...the mistake in the code applied to the program's evaluation of both its own moves and the opponent's possible moves. In other words the program assumed that the opponent was also playing to lose.

How do you play to lose a game of chess if your opponent also wants to lose? You need to get to a position where the only legal move of the opponent is to checkmate you.

You'll want a position where you have a big material advantage, and all the opponent has is their king and enough material to mate you. Probably just king and queen. Then you'd need to keep putting them in check, in such a way that they have to block with the queen. You'd need to arrange a series of such checks and blocks so that the final block also delivers checkmate on you.

And so it turns out that during the opening and middle game, playing to lose against someone who is also playing to lose looks pretty much the same as playing to win against someone who is also playing to win.

(Personally, I doubt this actually happened. The story is old, and I don't think chess programs would have been able to see far enough ahead for them to discover that getting an overwhelming position is the way to force the opponent to checkmate them).

Re: Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves

#29

Post by [deleted] »

908B64B197 wrote: post #3

I recall an AI playing a game that, just before losing, would press the "menu" button to pause the game and then stop doing anything. It fulfilled it's goal (not lose). That behavior should set expectations when it comes to AI. Also, am-I the only one that thinks this whole "chat bot having a natural conversation to book an appointment" is useless when a simple date-picker would do?

[deleted]
Post reply on HN