Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI tools have been built to catch Covid, but none of them helped

technologyreview.com

Post reply on HN
Print view View original

1–10 of 136 posts

Re: AI tools have been built to catch Covid, but none of them helped

#2

Post by vitorsr »

> “The models are so similar—they almost all use the same techniques with minor tweaks, the same inputs—and they all make the same mistakes,” says Wynants. “If all these people making new models instead tested models that were already available, maybe we’d have something that could really help in the clinic by now.”

Sounds about right.

Re: AI tools have been built to catch Covid, but none of them helped

#5

Post by new299 »

> Many unwittingly used a data set that contained chest scans of children who did not have covid as their examples of what non-covid cases looked like. But as a result, the AIs learned to identify kids, not covid.

> Driggs’s group trained its own model using a data set that contained a mix of scans taken when patients were lying down and standing up. Because patients scanned while lying down were more likely to be seriously ill, the AI learned wrongly to predict serious covid risk from a person’s position.

> Errors like these seem obvious in hindsight. They can also be fixed by adjusting the models, if researchers are aware of them.

I wouldn't trust any method of fixing errors in data collection. You need to redo data collection and perform a proper case-control study. You need to select your cases and controls such that the demographics look similar.

In general, your goal is not to pick cases out against a background of the general population. It's to pick cases out from an at risk group.

And if there's any difference in procedure used for data collection between cases and controls, your data is compromised. Researchers often suggest correcting the dataset (cropping out artifacts etc.). In my opinion this doesn't really work... artifacts (for example in image data) can often cause subtile global differences in illumination which ML will pick up on but not be obvious to someone inspecting the images.

The only solution when you have such artifacts (and only one that would be acceptable to me if I was doing DD, or evaluating the research) is to redo data collection in the context of a more reasonable case-control study.

Re: AI tools have been built to catch Covid, but none of them helped

#6

Post by briefcomment »

vitorsr wrote: post #2

> “The models are so similar—they almost all use the same techniques with minor tweaks, the same inputs—and they all make the same mistakes,” says Wynants. “If all these people making new models instead tested models that were already available, maybe we’d have something that could really help in the clinic by now.” Sounds about right.

I'm curious what people find is more likely given this evidence:

- 100s of models were done incorrectly, and none even remotely correctly

- there is no phenomenon to correctly model

Both seem extremely unlikely. Not sure what the alternatives are.

Re: AI tools have been built to catch Covid, but none of them helped

#9

Post by tablespoon »

briefcomment wrote: post #6
vitorsr wrote: post #2

> “The models are so similar—they almost all use the same techniques with minor tweaks, the same inputs—and they all make the same mistakes,” says Wynants. “If all these people making new models instead tested models that were already available, maybe we’d have something that could really help in the clinic by now.” Sounds about right.

I'm curious what people find is more likely given this evidence: - 100s of models were done incorrectly, and none even remotely correctly - there is no phenomenon to correctly model Both seem extremely unlikely. Not sure what the alternatives are.

That's a false choice. There's also:

- Technology not suitable or adequate for this use case.

I mean, we've been to the "AI over-promises and under-delivers" rodeo before.

Post reply on HN