Why Can't Epic's AI model Detect Sepsis? • Hacker News Board

Re: Why Can't Epic's AI model Detect Sepsis?

#11

Post by ashtonkem » Fri, Oct 15, 2021, 2:16 PM UTC

stuartbman wrote: post #9

ethanbond wrote: post #6

Earlier quoted context omitted.

Can you add more detail to this thought, other than just trying to vaguely affiliate proprietary tech with failure?

One of the problems with closed models is that any model can be found to train on the 'wrong' data point. So e.g. a chest x ray reader determines that images taken with the machine in ICU indicate sicker patients than elsewhere- that's not useful. If you can't inspect the model to check that, they might claim superior performance, but then the model doesn't work as well as advertised when it's tried out. Other biases…

One of my more radical opinions in this area is the idea that it should be illegal to sell a closed and proprietary ML model for areas of public safety, specifically in hospitals and in courts/jails. The public’s interest in transparency in such matters trumps the company’s copyrights. Trained experts get a chance to inspect every drug and every medical device that’s used; why shouldn’t they get to see how a ML model used in a hospital was trained?

Re: Why Can't Epic's AI model Detect Sepsis?

#12

Post by stuartbman » Fri, Oct 15, 2021, 2:22 PM UTC

ashtonkem wrote: post #11

stuartbman wrote: post #9

Earlier quoted context omitted.

One of the problems with closed models is that any model can be found to train on the 'wrong' data point. So e.g. a chest x ray reader determines that images taken with the machine in ICU indicate sicker patients than elsewhere- that's not useful. If you can't inspect the model to check that, they might claim superior performance, but then the model doesn't work as well as advertised when it's tried out. Other biases…

One of my more radical opinions in this area is the idea that it should be illegal to sell a closed and proprietary ML model for areas of public safety, specifically in hospitals and in courts/jails. The public’s interest in transparency in such matters trumps the company’s copyrights. Trained experts get a chance to inspect every drug and every medical device that’s used; why shouldn’t they get to see how a ML model…

Completely agree. I've not seen it tested legally, but the EU now has a 'right to explanation' where automated decisions are made about people. This would prohibit closed ML from most arenas.

Re: Why Can't Epic's AI model Detect Sepsis?

#13

Post by amelius » Fri, Oct 15, 2021, 2:50 PM UTC

stuartbman wrote: post #12

ashtonkem wrote: post #11

Earlier quoted context omitted.

One of my more radical opinions in this area is the idea that it should be illegal to sell a closed and proprietary ML model for areas of public safety, specifically in hospitals and in courts/jails. The public’s interest in transparency in such matters trumps the company’s copyrights. Trained experts get a chance to inspect every drug and every medical device that’s used; why shouldn’t they get to see how a ML model…

Completely agree. I've not seen it tested legally, but the EU now has a 'right to explanation' where automated decisions are made about people. This would prohibit closed ML from most arenas.

But wouldn't that cause problems with disclosure of patient data?

I.e., if you want to explain ML decisions, you'd have to provide the training data, which is sensitive data.

Re: Why Can't Epic's AI model Detect Sepsis?

#14

Post by ashtonkem » Fri, Oct 15, 2021, 2:56 PM UTC

amelius wrote: post #13

stuartbman wrote: post #12

Earlier quoted context omitted.

Completely agree. I've not seen it tested legally, but the EU now has a 'right to explanation' where automated decisions are made about people. This would prohibit closed ML from most arenas.

But wouldn't that cause problems with disclosure of patient data? I.e., if you want to explain ML decisions, you'd have to provide the training data, which is sensitive data.

Seems like there should be some informed consent before your identifiable medical data is used for ML training then.

Re: Why Can't Epic's AI model Detect Sepsis?

#15

Post by tboyd47 » Fri, Oct 15, 2021, 3:23 PM UTC

stuartbman wrote: post #2

Granted this is the review of a single model, but given that it's by one of the big players in the EMR space, it shows how challenging this problem is. Sepsis is easy to under-recognise, and difficult to identify reliably without lots (and lots) of false alarms.

How challenging it is, and how easy it is to sell a product that apparently does not meet the challenge.

Re: Why Can't Epic's AI model Detect Sepsis?

#16

Post by stuartbman » Fri, Oct 15, 2021, 3:51 PM UTC

amelius wrote: post #13

stuartbman wrote: post #12

Earlier quoted context omitted.

Completely agree. I've not seen it tested legally, but the EU now has a 'right to explanation' where automated decisions are made about people. This would prohibit closed ML from most arenas.

But wouldn't that cause problems with disclosure of patient data? I.e., if you want to explain ML decisions, you'd have to provide the training data, which is sensitive data.

Yes this is why I'm not sure how it's been tested. In the scientific literature in many cases the data is anonymised and made available publicly or to those interested, but you can't always anonymise the data adequately, so an audit process might be necessary

Re: Why Can't Epic's AI model Detect Sepsis?

#17

Post by telchar » Fri, Oct 15, 2021, 4:12 PM UTC

mdaidc wrote: post #7

The title is misleading and wrong. "AI" from a specific company(Epic) can't detect sepsis. But technology exists from people that known what they are doing, such as Johns Hopkin's spin-off Bayesian [1], and they have been very successful in detecting Sepsis. [1] https://www.bayesianhealth.com

Epic would tell you that they have been very successful in predicting sepsis too. But it's hard to train a model that generalizes to data from other sources (the problem described in the article). How does BH's model work outside Johns Hopkins? That's the real test.

Re: Why Can't Epic's AI model Detect Sepsis?

#18

Post by amelius » Fri, Oct 15, 2021, 4:18 PM UTC

Enginerrrd wrote: post #4

I used to work in emergency medicine as a paramedic. A far cry from an MD for sure, but here's my two cents: Unless they were already in decline, I never suspected Sepsis based solely on vitals. It was usually more of a gut feel thing. It can present with weird symptoms, like unexplained joint pain in multiple joints and things like that. I think that two key tells are patient history and patient cognition relative t…

> "How are they normally?" It's hard to measure these things with vitals so the AI doesn't have the right data going in, IMO.

Perhaps a task for a smartwatch?

Re: Why Can't Epic's AI model Detect Sepsis?

#19

Post by telchar » Fri, Oct 15, 2021, 4:18 PM UTC

amelius wrote: post #13

stuartbman wrote: post #12

Earlier quoted context omitted.

Completely agree. I've not seen it tested legally, but the EU now has a 'right to explanation' where automated decisions are made about people. This would prohibit closed ML from most arenas.

But wouldn't that cause problems with disclosure of patient data? I.e., if you want to explain ML decisions, you'd have to provide the training data, which is sensitive data.

I don't see why the training data would need to be provided. The model would, and that's derived from the training data, but the training data itself shouldn't need to be provided. It is hard to explain a model with any degree of complexity with or without training data, so it might not be easy, but that's just the nature of complex models.