Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Garry Kasparov on IBM's Watson

theatlantic.com

Post reply on HN
Print view View original

1–10 of 24 posts

Re: Garry Kasparov on IBM's Watson

#3

Post by jbooth »

jhamburger wrote: post #2

Not quite clear on why people keep pointing to the 'Toronto' question as proof that Watson is fundamentally flawed in some irreconcilable way.

Yeah.. for all those people saying it's not impressive, I'm forced to wonder why they didn't just build it, then.

EDIT: Wow this comment is way more controversial than I thought when writing it. Down to -1, up to 2, back to 0.

Anyone who finds it so objectionable as to downrate, please explain why that's so? Discussion > downvoting.

Re: Garry Kasparov on IBM's Watson

#4

Post by kenjackson »

Worse, by definition they do not understand what they do not understand and so cannot avoid them

Kasparov didn't seem to see what I did. Watson seemed very consistent in knowing what it did not know. There was maybe two questions I recall where it actually got the question wrong with 50%+ certainty. I believe it answered, "leg" when it should have been "mising a leg". The other it answered the 20s when the answer was the 10s. And I think for neither of those the percentage was much beyond 50%.

Also Kasparov seems to miss that Watson in medicine would be used with humans. I doubt a doctor will say, "Watson says to cut off his left leg -- I would just given him aspirin for the headache, oh well. Hopefully cutting off this leg makes his head feel better."

What Watson hopefully will do is help diagnosis. Especially tricky ones.

There's a great story in a book I read, I wish I could recall the name, but it begins with a lady who has some stomach issue that she has for like 20 years. Everyone thinks its in her head. She finally happens upon a doctor who happens to have seen something like this before, she gets diagnosed and healed. But she had to live with it for like 20 years after seeing doctor after doctor. Watson would be able to greatly help situations like this, I hope.

UPDATE: The book is "How Doctor's Think". Here's an excerpt that talks about this case, http://harvardmedicine.hms.harvard.edu/bulletin/winter2007/7... -- just in case anyone cares. :-)

Re: Garry Kasparov on IBM's Watson

#5

Post by ugh »

jhamburger wrote: post #2

Not quite clear on why people keep pointing to the 'Toronto' question as proof that Watson is fundamentally flawed in some irreconcilable way.

Everyone seems to forget that Watson would never have answered Toronto if it didn’t have to. It wasn’t at all confident, you can’t even really say that it made a mistake. It just didn’t know the answer and was – correctly so – very sure that it didn’t know the answer.

Re: Garry Kasparov on IBM's Watson

#6

Post by Jach »

jhamburger wrote: post #2

Not quite clear on why people keep pointing to the 'Toronto' question as proof that Watson is fundamentally flawed in some irreconcilable way.

That's not at all what Kasparov said:

    My concern about its utility, and I read they would like it to answer medical questions, is that
    Watson's performance reminded me of chess computers. They play fantastically well in maybe 90% of
    positions, but there is a selection of positions they do not understand at all. Worse, by definition
    they do not understand what they do not understand and so cannot avoid them. A strong human Jeopardy! player,
    or a human doctor, may get the answer wrong, but he is unlikely to make a huge blunder or category error--
    at least not without being aware of his own doubts. We are also good at judging our own level of certainty.
    A computer can simulate this by an artificial confidence measurement, but I would not like to be
    the patient who discovers the medical equivalent of answering "Toronto" in the "US Cities" category,
    as Watson did.
    
    I would not like to downplay the Watson team's achievement, because clearly they did something most
    did not yet believe possible. And IBM can be lauded for these experiments. I would only like to wait
    and see if there is anything for Watson beyond Jeopardy!.
If IBM wants to fix the "Toronto" problem, have at it. But those sorts of "embarrassing" errors could be quite costly in medical situations. During the show they showed Watson's progression from really stupid answers very frequently to less frequently, which makes me personally believe their fundamental process is flawed (not necessarily irreconcilable) and their current algorithms are just a bunch of hacks thrown together on top of Google rather than something more sophisticated like Wolfram Alpha.

Re: Garry Kasparov on IBM's Watson

#7

Post by zwieback »

kenjackson wrote: post #4

Worse, by definition they do not understand what they do not understand and so cannot avoid them Kasparov didn't seem to see what I did. Watson seemed very consistent in knowing what it did not know. There was maybe two questions I recall where it actually got the question wrong with 50%+ certainty. I believe it answered, "leg" when it should have been "mising a leg". The other it answered the 20s when the answer was…

Totally agree. It's shocking how many doctors seem to be lacking basic knowledge about things like drug sideeffects or lesser known symptoms about common diseases. You obviously still need both specialists and general practitioners but I think it's time to revisit the idea of a medical expert system .

Re: Garry Kasparov on IBM's Watson

#8

Post by kokofoo »

jhamburger wrote: post #2

Not quite clear on why people keep pointing to the 'Toronto' question as proof that Watson is fundamentally flawed in some irreconcilable way.

I actually told my wife (who knew the correct answer - Chicago) half jokingly that perhaps it was the sportsmanship built into Watson to throw away an easy answer once in a while, because he was on a hot streak up to that point.

Re: Garry Kasparov on IBM's Watson

#9

Post by Jach »

ugh wrote: post #5
jhamburger wrote: post #2

Not quite clear on why people keep pointing to the 'Toronto' question as proof that Watson is fundamentally flawed in some irreconcilable way.

Everyone seems to forget that Watson would never have answered Toronto if it didn’t have to. It wasn’t at all confident, you can’t even really say that it made a mistake. It just didn’t know the answer and was – correctly so – very sure that it didn’t know the answer.

Excellent point. But that "Toronto" was its highest guess makes you wonder if it could confidently select something that exploits whatever errors led to Toronto. And if you asked it a medical question, would it say "I don't know?" if it wasn't confident or would it give an answer anyway (along with a confidence level that might be ignored or rationalized away)?

Re: Garry Kasparov on IBM's Watson

#10

Post by michaelcampbell »

Jach wrote: post #6
jhamburger wrote: post #2

Not quite clear on why people keep pointing to the 'Toronto' question as proof that Watson is fundamentally flawed in some irreconcilable way.

That's not at all what Kasparov said: My concern about its utility, and I read they would like it to answer medical questions, is that Watson's performance reminded me of chess computers. They play fantastically well in maybe 90% of positions, but there is a selection of positions they do not understand at all. Worse, by definition they do not understand what they do not understand and so cannot avoid them. A strong…

> but I would not like to be the patient who discovers the medical equivalent of answering "Toronto" in the "US Cities" category, as Watson did.

Surprise, that kind of mistake happens far too frequently in the medical field now.

Why is Kasparov commenting on something so far out of his recognized area of expertise relevant anyway? I don't go to Knuth for advice on chess, nor Hawking for snarky banter on economics, etc. (Although if I had access to either of those 2, I might try it.)

Post reply on HN