To me, the main mistake was the series of commercials giving the strong impression that IBM already had this incredible Artificial General Intelligence that was indistinguishable from a highly intelligent human and was solving a myriad of difficult practical problems better than any expert. I suspect that most who were well-versed in AI felt the ads were disingenuous from the start. I know I did. I think the marketing campaign would have better served IBM had it laid out their commitment to achieve these things without sounding like they were already there.
IBM Watson Overpromised and Underdelivered on AI Health Care
21–30 of 127 posts
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#22What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?
Either the AI can generalize or it can't. If it can't, you don't need neural nets at all. If it can, you need to worry about whether it's generalizing correctly, which is definitely "so hard".
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#23I worked on this project and there were a lot of issues. Two of the biggest were:
* Whatever the quality of the technology (which I personally never saw as that compelling) was wrapped up in terribly written research code, making it practically impossible to setup and use.
* The Jeopardy demo was made possible by the existence of a marked-up source of general knowledge (Wikipedia), a ready-made bank of questions and answers from past shows (j-archive.org), and the fact that practically anyone had the ability to curate more Q&A pairs. This is almost totally different than the medical use case where the knowledge is wrapped up in proprietary textbooks and papers and the only people able to curate training data are medical professionals.
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#24>”But Watson won’t change its conclusions based on just four patients. To solve this problem, the Sloan Kettering experts created “synthetic cases” that Watson could learn from, essentially make-believe patients with certain demographic profiles and cancer characteristics.” Is this standard practice in machine learning? This sounds more like regular programming to get exactly the outcome you want.
It's standard practice for idiots. And apparently "Sloan Kettering experts."
There are semi-supervised techniques to do stuff like this in a more systematic/automated way, but you still don't get anything for free: the outcome depends on the priors used to do the semi-supervised voodoo. In a generous moment I might assume this is what they meant, but it's still dumb.
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#25What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?
[deleted]
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#26What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?
Sparse data. Neural nets (for example) do horribly at this sort of thing; they're data hungry.
It's super difficult to beat something like linear regression in this sort of thing (ideally combined with domain knowledge -something neural approaches mostly fail at), and even linear regression gives awful results.
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#27Earlier quoted context omitted.
To be fair, IBM has succeeded at big jobs too. For instance, they developed software for the Apollo mission and the Space Shuttle. The IBM/360. The IBM PC. AS/400.
I don't think there's anyone who disagrees that IBM was a great company once. Do you have a more recent example? something that happened several years after Louis Gerstner first assumed leadership.
They are really good at cashing checks and charging more on top of that: http://www.investigativepost.org/2018/02/19/taxpayers-fund-i...
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#28What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?
There are huge barriers to getting a healthcare AI product to market in the United States. First off, data is generally extremely difficult and expensive to gather due to the need to protect patient info according to the Health Insurance Portability and Accountability Act (HIPAA). To get representative data, you need to gather from many sites and the data needs to be anonymized and "truthed" by a medical expert who typically must examine multiple medical reports and case history for each patient. Each data collection site requires approval for data collection by an Institutional Review Board, which may take months or even years to get. Assuming you are able to get a reasonable amount of data for development and testing, you need to solve the actual AI problem (probably very difficult due to the extremely heterogeneous nature of disease you are dealing with) and then you need regulatory approval. Depending on the FDA classification of your product, this can be another many-month to many-year process of data collection and clinical studies.
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#29What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?
The problem is that the numerous easy cases do not result in a useful network ... any 1st-year intern will get the easy results already.
The hard cases, which would be useful to a doctor, occur very rarely. My father's unusual reaction to a post-bypass drug regimen was something like the 3rd time that happened in Canada. How do you "train" that into a neural network?
Re: IBM Watson Overpromised and Underdelivered on AI Health Care
#30Earlier quoted context omitted.
I don't think there's anyone who disagrees that IBM was a great company once. Do you have a more recent example? something that happened several years after Louis Gerstner first assumed leadership.
They're still solid in the supercomputer space, no? BlueGene, and Summit and Sierra more recently are IBM projects. Admittedly though given their pretty large size I can't name much else.
> Admittedly though given their pretty large size I can't name much else.
Exactly. Even if 20,000 people worked on those supercomputers, mainframes and Watson, what do the other 350,000 employees work on? Consulting, and it's been that way since Gerstner.