IBM's Watson AI trumps humans in "Jeopardy!"

Watson will go against former best "Jeopardy!" players this Fall. However, I think it's a bit overarching to label this as an application of a natural question answering system, as is done in this article. "Jeopardy!"'s answer snippets are not like normal questions that, say, a tax software would encounter, I think.

"When one I.B.M. executive suggested taking on “Jeopardy!” he was immediately pooh-poohed. Deep Blue was able to play chess well because the game is perfectly logical, with fairly simple rules; it can be reduced easily to math, which computers handle superbly. But the rules of language are much trickier. At the time, the very best question-answering systems — some created by software firms, some by university researchers — could sort through news articles on their own and answer questions about the content, but they understood only questions stated in very simple language (“What is the capital of Russia?”); in government-run competitions, the top systems answered correctly only about 70 percent of the time, and many were far worse. “Jeopardy!” with its witty, punning questions, seemed beyond their capabilities. What’s more, winning on “Jeopardy!” requires finding an answer in a few seconds. The top question-answering machines often spent longer, even entire minutes, doing the same thing."

Still, it's a stunning achievement. And IBM will definitely recoup all the millions it put into this project from the PR money it has saved. Other big tech companies, take note please.

As a person who played Quiz Bowl in high school, it was pretty apparent that it was domain that would be mastered by AI in time.

Answering questions correctly was often a function of how much material you had studied and were able to recall. I get the feeling that the computer might have a slight advantage at pure recall but a disadvantage at associative recall. Even so, the sheer speed of computation from the computer far outpaces a human's ability to slam down on a buzzer.

On an unrelated note, this is beginning to more closely resemble the prophetic computers characteristic of Asimov stories.

In 2006, I was working at a startup that was trying to nail this space. It's amazing how far this sort of tech has come.

But in some ways, Jeopardy is much more constrained than tax questions. You know that the question will have a relatively short statement, the answer will be in the form of a question with probably no more than 5 words being actually relevant with the rest being used to make it into a question.

Furthermore, you know that breadth of knowledge is generally more significant depth. Having a database of every nation's capital and a few significant things about it is most likely more useful than being able to provide an in depth discussion of quantum electrodynamics.

Tax questions on the other hand often have lengthy and detailed statements and require an essay style answer. Worse, in some cases, detailed tax advice may actually require judgement and advice. Of course, tax software takes short cuts in this respect. It is not designed to handle complicated situations with nuances. It is designed to handle your average consumer, and even there is constrains the problem by being the one that asks questions and then producing forms rather than answering ad hoc questions.

(edit: fixed grammar)

You can play against Watson online: http://www.nytimes.com/interactive/2010/06/16/magazine/watso...