AI tools have been built to catch Covid, but none of them helped • Hacker News Board

Re: AI tools have been built to catch Covid, but none of them helped

#121

Post by dontreact » Sat, Jul 31, 2021, 7:42 PM UTC

iamstupidsimple wrote: post #71

dontreact wrote: post #23

The silly thing here is that it’s very easy to train models nowadays, so the medical AI space is very noisy. With the right data and the right application, this technology can definitely be helpful. However, it is very hard to sort out signal from noise since any reasonably competent undergrad can throw a convnet at some images and get an AUC. Actually, I have even seen a high school science fair where they did a med…

I'm not sure if this has been attempted, so please do correct me, but I would very much like to see AI models designed to assist rather than give coarse classifications like "X disease detected". The best example I can think of would be creating bounding boxes around specific tells, or listing specific markers that a clinician would be looking for themselves, leaving the final decision up to them.

Nearly all FDA approvals for medical imaging diagnostic software are for programs that assist. It’s been done for decades.

Re: AI tools have been built to catch Covid, but none of them helped

#122

Post by dontreact » Sat, Jul 31, 2021, 7:44 PM UTC

Jweb_Guru wrote: post #87

Grimm1 wrote: post #30

Earlier quoted context omitted.

Why not? Plenty of researchers aren’t thorough. I’m not saying that’s the case here but it’s entirely plausible from what I know about academia and wouldn’t be a crazy claim to make.

If a model did particularly well while almost every other model did badly, it would have gained attention in the community (not to mention, the researchers who worked on it would be actively marketing this!). Even if the researchers who worked on the paper weren't aware of it, it would almost certainly be brought up during peer review. The only way that such a model would not end up on such an extensive list is delib…

I don’t think this is true. Medical practices change slowly and it takes time to build trust. It’s possible a covid model will become clinically useful, but it probably requires having enough buy in to run a clinical trial and then for the economics to line up so that using it leads to more money for hospitals somehow

Re: AI tools have been built to catch Covid, but none of them helped

#123

Post by dontreact » Sat, Jul 31, 2021, 7:46 PM UTC

mattnewport wrote: post #28

dontreact wrote: post #23

The silly thing here is that it’s very easy to train models nowadays, so the medical AI space is very noisy. With the right data and the right application, this technology can definitely be helpful. However, it is very hard to sort out signal from noise since any reasonably competent undergrad can throw a convnet at some images and get an AUC. Actually, I have even seen a high school science fair where they did a med…

> I am reasonably confident that there is a group out there that was more careful and has something that might have been useful What is that confidence based on? It seems like the researchers here made a reasonably thorough effort to find "something that might have been useful" and only turned up two that were even worth further investigation: "She and her colleagues have looked at 232 algorithms for diagnosing patie…

I think the issue is not with the training step but with the validation step. The only way to build enough trust is to carefully run a prospective clinical trial, and before that you probably need enough retrospective evidence to get some sort of FDA approval.

These processes take a lot of time and it’s not surprising that a year and a half after COVID-19 became widespread in the US, we still don’t have a fully validated model.

Re: AI tools have been built to catch Covid, but none of them helped

#124

Post by dontreact » Sat, Jul 31, 2021, 7:48 PM UTC

dontreact wrote: post #123

mattnewport wrote: post #28

Earlier quoted context omitted.

> I am reasonably confident that there is a group out there that was more careful and has something that might have been useful What is that confidence based on? It seems like the researchers here made a reasonably thorough effort to find "something that might have been useful" and only turned up two that were even worth further investigation: "She and her colleagues have looked at 232 algorithms for diagnosing patie…

I think the issue is not with the training step but with the validation step. The only way to build enough trust is to carefully run a prospective clinical trial, and before that you probably need enough retrospective evidence to get some sort of FDA approval. These processes take a lot of time and it’s not surprising that a year and a half after COVID-19 became widespread in the US, we still don’t have a fully valid…

In fact the claim the paper makes is that no model has been sufficiently clinically validated.

Problem is: how do you get enough buy in from clinicians to run a clinical trial with YOUR model, when there are hundreds of crappy models out there? It takes a lot of time to run these studies for this reason to start with and then depending on the size of the clinical trial, it takes time for the trial to run it’s course.

Re: AI tools have been built to catch Covid, but none of them helped

#125

Post by parineum » Sat, Jul 31, 2021, 7:55 PM UTC

johnisgood wrote: post #102

parineum wrote: post #96

Earlier quoted context omitted.

I'm in California and I have the same experience as you but the data nationally shows otherwise.

Which one? I'm looking at https://www.kff.org/policy-watch/the-red-blue-divide-in-covi... at the moment. July 6 - 46.7% Biden, 35.0% Trump. This is reported by the way, right?

Doesn't that confirm what I said? That's, in Biden counties, 46.7% of people are vaccinated and in Trump counties, 35% of people are vaccinated.

Those numbers are vaccination rates in red vs. blue counties where the blue counties have higher rates than the red counties.

That interpretation is supported by the way the article is framing the data in the conclusion.

> Although there has been an overall significant slow-down in COVID-19 vaccination rates in the U.S., these findings show a widening divide of communities at risk for COVID-19 along partisan lines. A key component of any effort to boost vaccination rates among Republicans will be...

Re: AI tools have been built to catch Covid, but none of them helped

#126

Post by parineum » Sat, Jul 31, 2021, 8:05 PM UTC

TomSwirly wrote: post #108

parineum wrote: post #75

Earlier quoted context omitted.

I've been telling people that the way to get people to take the vaccine would be to give Trump credit for operation lightspeed (which I think he deserves but that's irrelevant) and start referring to the vaccine as Trump's Vaccine. I'm okay with handing Trump a win if it can help save lives and end this pandemic.

> give Trump credit for operation lightspeed Your idea wouldn't work. MAGAs hate liberals, and whatever liberals say appears to them like the voices of the grown-ups in the Peanuts cartoons. Maybe they pull out a few phrases to parody. We've had over a year of near-continuous propaganda from the Republicans. Masks threaten your freedom. COVID is just the flu. No one dies from it, they put it down on every death certi…

Is Trump not responsible for Operation Warp speed? The program that spent billions of dollars to stockpile untested vaccines while they were going through testing so that we'd have an ample supply to distribute in the event they were approved (on an accelerated time scale)?

Operation Warp speed is why, while everyone else is still struggling with supply, everyone the US who wants a vaccine can get a vaccine.

https://en.wikipedia.org/wiki/Operation_Warp_Speed

Unsurprisingly, the Biden administration did the exact opposite of what I suggested. Further entrenching COVID as a partisan issue.

> In January 2021, White House press secretary Jen Psaki announced that the program was expected to undergo a restructure and renaming under the Biden administration. Also in January 2021, Dr. Moncef Slaoui, former Operation Warp Speed lead, was told not to use the name Operation Warp Speed anymore.

Re: AI tools have been built to catch Covid, but none of them helped

#127

Post by nradov » Sat, Jul 31, 2021, 9:58 PM UTC

kQq9oHeAz6wLLS wrote: post #37

jauer wrote: post #34

Earlier quoted context omitted.

(also frustrated) re: adboys and girls, the problem (at least in the US) is that vaccines are political, so persuading people to get them is seen as dangerously close to subverting democratic society. Same thing as taking action to counter foreign influence campaigns. see also: the leaks to project veritas about Facebook's counter vaccine hesitancy program.

Well, back when Harris and Biden both said they were hesitant to get a vaccine produced under Trump, they kinda set the stage for the dumpster fire we have today.

Several state governments including California stated that they wouldn't trust FDA vaccine approvals and wanted to do their own safety reviews. That didn't help with public confidence either.

https://www.gov.ca.gov/2020/10/27/western-states-join-califo...

Re: AI tools have been built to catch Covid, but none of them helped

#128

Post by nradov » Sat, Jul 31, 2021, 10:03 PM UTC

selimthegrim wrote: post #93

Leparamour wrote: post #74

Earlier quoted context omitted.

>The companies delayed their announcement and strengthened the trials so the current Trumpista favorite argument would in theory lose steam. They [Pfizer] delayed the announcement to three days after the elections. I'm sure those three days were critical in testing and making the vaccine safe for the consumer./s It was an obvious political move to deny back-then US president Trump a public success as a consequence fr…

I live in New Orleans where unvaccinated cases are exploding and people in surrounding areas specifically cite Democrats as a reason not to take the vaccine. I’ll stand by my words. And I’m not fan of Pfizer trying to make vaccine policy by press release but didn’t one of the companies turn down Warp Speed money to avoid this perception?

All of the vaccines used in the US received funding through Operation Warp Speed. J&J and Moderna got R&D money and production orders. Pfizer only got production orders.

https://en.wikipedia.org/wiki/Operation_Warp_Speed?wprov=sfl...

Re: AI tools have been built to catch Covid, but none of them helped

#129

Post by nradov » Sat, Jul 31, 2021, 10:08 PM UTC

29athrowaway wrote: post #72

Superspreaders do not care about compliance or COVID prevention tools. Superspreaders (antimaskers, antilockdown, antidistancing, antivaxxers) are the people keeping COVID alive.

SARS-CoV-2 is now endemic in the worldwide human population, just like several other coronaviruses. It will never go away regardless of what people do. Fortunately the vaccines can limit the death toll.

Re: AI tools have been built to catch Covid, but none of them helped

#130

Post by nradov » Sun, Aug 01, 2021, 1:05 AM UTC

bumby wrote: post #120

nradov wrote: post #20

What ever happened to using AI to diagnose infections based on cough sounds? It seemed like a promising approach. https://news.mit.edu/2020/covid-19-cough-cellphone-detection...

I vaguely remember reading more about this elsewhere and I believe they only used COVID positive patients. It would be interesting if there are follow-up studies with mixed training data if true.

No the researchers tested using both positive and negative patients.