Garry Kasparov on IBM's Watson

In the medical case, it's actually better for the answer to be obviously, embarrassingly wrong than slightly wrong. Like the other commenter said, people aren't going to be getting amputations for headaches just because Watson says so. There's much more danger in something like prescribing medications with a fatal interaction, something that a hypothetical "Dr. Watson" would pick up.

"As with Deep Blue, he had once again let an encounter with a machine play games with his head. He had been obsessed with the idea that Deep Junior would never tire. 'The machine is never distracted by an argument with its mother," he told me, 'or a lack of sleep.'

And in the linked piece Kasparov alludes to the reported next approach IBM wants to take with Watson - support in medicine.

Kasparov's human reaction to his encounters with Watson's distant cousins brings up one obvious benefit in the use of technology like Watson for supporting medical decision-making - simply that such software will be less likely to miss something. Software is less likely to miss considering a diagnosis, ordering a crucial test, or following up on a finding - unlike the fallible 'I' who may have skipped a class in med school, or was up all night on call and just can't think straight, or am just occasionally more stupid than usual.

Diagnosis is the first thing people think of with technology like this, but in my opinion that's not the big problem Watson should tackle. Medical diagnosis in and of itself (dramatizations like the TV show 'House' notwithstanding), is not really that difficult 99% of the time. When you hear hoofbeats, you're very likely going to find horses and not zebras. A future Dr. Watson might occasionally be very helpful in pointing out very obscure (but uncommon) diagnoses. However, in my opinion the most helpful thing a Dr. Watson could provide is collecting, evaluating, and comparing evidence and outcomes as they are developed globally and locally (ie across broad swaths of medicine, but also within a single physician's own patient population), continuously educating the physician, and monitoring cases.

There is plenty of untapped medical data/evidence out there, but it's almost all hidden away in plain sight...text/natural language. I have to agree with Kasparov here, in that the primary advancement Watson represents was in moving farther down the path from syntax to semantics.

Instead, IBM wanted a forum to show off its multi-million-dollar QA technology, and approached Jeopardy. (They may have also, though I haven't seen definitive information either way, offered Jeopardy promotional payments.) IBM then spent 3+ years optimizing for the Jeopardy domain. (In the Reddit QA, the Watson team answered: "At this point, all Watson can do is play Jeopardy and provide responses in the Jeopardy format.")

And in the matches, Watson dominated on one dimension of Jeopardy play – quickly pressing a button after a light goes off – that's the least interesting technical challenge. (Yes, it's an important part of any champion's skills, but a machine would have won that button-pressing competition 50 years ago, so it obscures rather than highlights any other 'breakthroughs' Watson may represent.)

While impressive in several dimensions, and drawn from much deeper research by IBM, the only thing we can say for sure about Watson is that it was a "Horse for the Course" in Jeopardy. And unfortunately, no other computer horses were invited to play, and offered the same prizes (in money and fame).

I suspect, now that the pattern has been set, we'll see leaner teams showing they can do as well or better than Watson with far less funding/hardware, over the next few years. Still, in the popular imagination, these efforts will live in the shadow of Watson, when a fair competitive process might have given them a chance to upstage Watson.

Agree on your suspicion. Simply quartering the cost of memory and copying the approach from the paper with some home-grown improvements will get people ahead of IBM and probably inside IBM's decision loop so they're permanently ahead. But plowing something the first time is often the hardest. These weren't dumb people working on this thing for 3+ years.

The second reason is that IBM was representing Watson as something of a big push in knowledge representation (I just watched a video where they talk about Watson's "informed judgments" about complicated questions for instance). It looks instead like Watson just has an improved ability to disambiguate words relative to previous systems and to do quick lookups that match those words with nearby key terms.

For example, on the clue "Rembrandt's biblical scene 'Storm on the Sea of' this was stolen from a Boston museum in 1990", Watson correctly answered "Galilee". But its next two answers were "Gardner Museum" and "Art theft"; no one who "understood" the question in any conventional sense would even consider these as answers because they don't make any sense. Clearly, Watson looked for instances of "Rembrandt", "Storm on the sea of", "stolen", or other phrases from the clue in its text corpus, and found that "Galilee", "Gardner Museum", and "art theft" all frequently occurred when together (because the painting was stolen from the Gardner museum in an instance of art theft), and relatively rarely when not together. "Galilee" probably won out of these three because Watson is tuned to Jeopardy clue styles (whenever there is a quoted phrase in a clue followed by the word 'this', it's always asking for the answer that completes the phrase).

Similarly, Watson was far less confident on the clue "You just need a nap!" You don't have this sleep disorder that can make sufferers nod off while standing up." It still got the right answer of "Narcolepsy", but with a relatively low confidence of 64%. "Insomnia" had a confidence of 32% despite clearly being the opposite sort of sleep disorder, and "deprivation" appeared at 13%, despite not being a sleep disorder. Here Watson gets confused because the only term of the clue that appears more frequently with "narcolepsy" than "insomnia" is "standing up"; my guess is that if "standing up" had been replaced by some oddly phrased, uncommonly occurring synonym, Watson wouldn't have been able to come up with an answer, despite the clue conveying exactly the same information.

This kind of cleverness is certainly impressive, but it seems like it's an advance in tuning existing techniques to the format of Jeopardy, not an advance that will spark other successful projects down the line. IBM's goal of giving us "the computer from Star Trek" doesn't seem any closer; I don't see any evidence that Watson could have answered a question that required more thought or understanding than a simple text search. If there was the question "how many kings ruled England in between Henry the Fourth and Henry the Eigth" (8), then Ken and Brad would have been able to answer relatively easily, while my guess is that Watson would be stumped.