Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM Watson Overpromised and Underdelivered on AI Health Care

spectrum.ieee.org

Post reply on HN
Print view View original

101–110 of 127 posts

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#101

Post by 1e-9 »

Randypea wrote: post #99
1e-9 wrote: post #81

Earlier quoted context omitted.

Unfortunately, there are still many cancers detected far too late for effective treatment. It sounds like you were indeed fortunate to have a less aggressive form. AI for cancer screening generally falls under the category of "Computer-Aided Detection" or CAD. The commercial and academic CAD efforts tend to be organized by the primary anatomical site of cancer and the detection method (e.g. X-Ray, CT-Scan, PET, ultra…

My cancer was finally found in my intestine. This is another opportunity. My primary oncologist and local surgeon were telling me the primary tumor, which has metastasized to a very large liver tumor, could not be found. I did my own research and found they had ordered the wrong type of imaging scan. Only after I pushed to have the correct scan (Gallium 68 PET/CT scan) was the primary tumor found. This was a "lack of…

As we are getting a bit off-topic, I temporarily placed an email address in my profile, which you can use to contact me for further discussion.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#102

Post by mr_toad »

ethbro wrote: post #74
jackfoxy wrote: post #67

Earlier quoted context omitted.

My company does machine learning checked by physical models. Our single biggest problem (and management is finally waking up to it) is curating the incoming data. And this is in a mature industry (oil & gas).

My biggest surprise has been how little everyone is aware of their data quality. The only explanation I've been able to come up with is that when it's all human processed, Joe 2nd-link-in-the-chain just deals with all the inconsistencies as best he can to get his job done, and never reports issues up.

Without those data inconsistencies to fix up every week, Joe would probably be out of a job.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#103

Post by mr_toad »

qwerty456127 wrote: post #19

What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?

There are hundreds of thousands of diagnostic and treatment codes, and most patients will have several, or even dozens of codes for a presentation. Add in the patients age and other demographic variables, medical history, and family history, and you have more combinations of variables than patients.

Except for some really common problems, like cases of influenza, every patient is unique.

AI can be used to handle simpler problems - identifying patients at risk of repeat admissions, or flagging the likelihood of dehydration for example.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#104

Post by 1e-9 »

Randypea wrote: post #99
1e-9 wrote: post #81

Earlier quoted context omitted.

Unfortunately, there are still many cancers detected far too late for effective treatment. It sounds like you were indeed fortunate to have a less aggressive form. AI for cancer screening generally falls under the category of "Computer-Aided Detection" or CAD. The commercial and academic CAD efforts tend to be organized by the primary anatomical site of cancer and the detection method (e.g. X-Ray, CT-Scan, PET, ultra…

My cancer was finally found in my intestine. This is another opportunity. My primary oncologist and local surgeon were telling me the primary tumor, which has metastasized to a very large liver tumor, could not be found. I did my own research and found they had ordered the wrong type of imaging scan. Only after I pushed to have the correct scan (Gallium 68 PET/CT scan) was the primary tumor found. This was a "lack of…

There currently is no good candidate for a imaging modality that can be used for a general screening program to find the top 15 to 20 cancers and I am unaware of anything on the near horizon. Such a scan would have to examine the neck thru the groin area to cover even just 10 out of the top 15 or so cancer types. Since screening involves patients with no symptoms, most patients won't actually have any disease and thus the imaging must be inexpensive, must have high sensitivity, must have a reasonable false positive rate, must involve little to no radiation, and must not require injection of contrast agents or radioactive tracers. That eliminates all of the imaging modalities I can think of that can examine large areas of the body for cancer. The best we have today are compromises on these criteria for patients that are at relatively high risk, such as a smoker or a cancer survivor, or for highly focused screening programs such as what we have for breast cancer.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#105

Post by ethbro »

mr_toad wrote: post #102
ethbro wrote: post #74

Earlier quoted context omitted.

My biggest surprise has been how little everyone is aware of their data quality. The only explanation I've been able to come up with is that when it's all human processed, Joe 2nd-link-in-the-chain just deals with all the inconsistencies as best he can to get his job done, and never reports issues up.

Without those data inconsistencies to fix up every week, Joe would probably be out of a job.

But Joe typically hates dealing with the inconsistencies, and he can tell you exactly how they could be fixed.

It generally seems like (a) the suggestions for fixes are impractical to implement (overly detrimental effect on counterparty), (b) Joe isn't empowered organizationally to suggest fixes that will be implemented, or (c) Joe doesn't have access to the IT tools to implement fixes himself.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#106

Post by mr_toad »

ethbro wrote: post #105
mr_toad wrote: post #102

Earlier quoted context omitted.

Without those data inconsistencies to fix up every week, Joe would probably be out of a job.

But Joe typically hates dealing with the inconsistencies, and he can tell you exactly how they could be fixed. It generally seems like (a) the suggestions for fixes are impractical to implement (overly detrimental effect on counterparty), (b) Joe isn't empowered organizationally to suggest fixes that will be implemented, or (c) Joe doesn't have access to the IT tools to implement fixes himself.

From my experience the answer is usually (d) all of the above.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#107

Post by djakjxnanjak »

AlotOfReading wrote: post #93
djakjxnanjak wrote: post #85

Earlier quoted context omitted.

If you can tune the specifics-sensitivity curve, you should be able to handle both cases. You have to be willing to refuse to provide an answer when you have low confidence.

The existence of adversarial attacks with high confidence on virtually all production ML systems should indicate that confidence numbers are not enough to rely on.

Do these attacks require fine control over the input? Eg. if you are scanning a patient’s body, does it matter if the model can be fooled by editing the values of individual pixels? This implies that you have a threat model where the data coming from the sensor is being manipulated, in which case all bets are off (the image could be entirely replaced). It doesn’t seem much different from a statistical model that you can blow up by feeding in values designed to cause a divide by zero (values that wouldn’t appear in real-world data).

It seems like a problem when classifying user-provided images (eg. identifying obscene images on a social network) but not so relevant when you own the sensors.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#108

Post by newen »

Main problem is Watson its like a million different things all with the same name. So no one actually knows what they are selling when they talk to you about Watson. Literally I had one IBM guy ask to check out their Watson and it took 5 minutes of them showing me the demo before I figured out what it was (cloud based Nvidia Digits alternative). If that particular Watson had an actual name, I probably would've heard about it before and I'd actually be able to recommend it to someone.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#109

Post by RichardCA »

hopler wrote: post #65
1e-9 wrote: post #21

To me, the main mistake was the series of commercials giving the strong impression that IBM already had this incredible Artificial General Intelligence that was indistinguishable from a highly intelligent human and was solving a myriad of difficult practical problems better than any expert. I suspect that most who were well-versed in AI felt the ads were disingenuous from the start. I know I did. I think the marketin…

IBM have been doing this since Deep Blue, if not older. It's their core business model: build cool tech demos and then sell consulting deployments that use nonexistent features of the tech

I think it runs deeper than that. I think the mythology of the "Electronic Brain" goes back to the 60's and all the Space Age idealism about technology.

Maybe I'm just naive, but I really think that strand still exists in IBM's corporate DNA. They still see this as solvable given sufficient time and effort.

Re: IBM Watson Overpromised and Underdelivered on AI Health Care

#110

Post by govg »

djakjxnanjak wrote: post #107
AlotOfReading wrote: post #93

Earlier quoted context omitted.

The existence of adversarial attacks with high confidence on virtually all production ML systems should indicate that confidence numbers are not enough to rely on.

Do these attacks require fine control over the input? Eg. if you are scanning a patient’s body, does it matter if the model can be fooled by editing the values of individual pixels? This implies that you have a threat model where the data coming from the sensor is being manipulated, in which case all bets are off (the image could be entirely replaced). It doesn’t seem much different from a statistical model that you…

You do have a point - all the adversarial attacks on ML models rely on full adversarial control on the inputs, which probably isn't the case with medical records. If there was unconstrained access to a patient's MRI scans by an adversary, then I don't think adversarial attacks on the ML diagnostic models are the biggest problems you'll face.
Post reply on HN